Jul 31 14:10:23.860253 ip-10-0-130-33 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 31 14:10:23.860268 ip-10-0-130-33 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 31 14:10:23.860278 ip-10-0-130-33 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 31 14:10:23.860600 ip-10-0-130-33 systemd[1]: Failed to start Kubernetes Kubelet. Jul 31 14:10:34.074338 ip-10-0-130-33 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 31 14:10:34.074355 ip-10-0-130-33 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fc6da98e613044568ae59c34cd418372 -- Jul 31 14:12:13.254327 ip-10-0-130-33 systemd[1]: Starting Kubernetes Kubelet... Jul 31 14:12:13.625890 ip-10-0-130-33 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:12:13.625890 ip-10-0-130-33 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 31 14:12:13.625890 ip-10-0-130-33 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:12:13.625890 ip-10-0-130-33 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 31 14:12:13.625890 ip-10-0-130-33 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 31 14:12:13.626756 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.626673 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 31 14:12:13.630973 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630958 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:13.630973 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630973 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630976 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630980 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630984 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630987 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630989 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630993 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630996 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.630998 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631012 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631015 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631018 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631021 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631024 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631027 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631029 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631032 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631034 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631037 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:13.631047 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631039 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631042 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631045 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631047 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631050 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631052 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631055 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631059 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631061 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631064 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631066 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631069 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631072 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631074 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631077 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631079 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631082 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631084 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631087 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:13.631501 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631089 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:13.631949 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.631092 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:13.632576 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632567 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632578 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632582 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632585 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632588 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632590 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632593 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632595 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632600 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632605 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632608 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632611 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:13.632610 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632614 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632617 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632619 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632622 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632625 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632627 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632630 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632632 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632636 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632640 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632642 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632645 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632647 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632651 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632653 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632656 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632659 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632661 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632663 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632666 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:13.632903 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632668 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632671 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632675 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632677 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632680 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632682 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632685 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632688 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632691 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632693 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632696 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632698 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632700 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632704 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632707 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632710 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632712 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632715 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632718 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632720 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:13.633406 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632723 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632726 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632728 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632731 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632733 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.632736 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633508 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633520 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633526 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633529 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633532 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633535 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633538 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633540 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633543 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633545 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633548 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633551 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633554 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633556 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:13.633878 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633559 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633561 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633563 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633566 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633569 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633571 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633575 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633577 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633580 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633583 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633585 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633588 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633590 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633593 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633595 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633597 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633600 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633602 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633606 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633616 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:13.634379 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633619 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633621 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633624 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633626 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633629 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633632 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633634 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633639 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633643 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633646 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633649 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633652 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633654 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633658 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633661 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633665 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633667 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633670 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633672 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:13.634873 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633675 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633678 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633680 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633684 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633687 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633689 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633692 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633694 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633697 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633699 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633703 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633706 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633710 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633718 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633721 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633724 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633727 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633730 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633732 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633735 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:13.635352 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633738 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633741 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633743 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633746 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633749 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633751 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633754 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633756 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633759 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633761 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633764 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633766 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633769 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633771 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633773 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633776 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633778 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633781 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633784 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633786 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:13.635858 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633789 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633791 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633793 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633796 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633799 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.633802 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634181 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634187 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634190 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634192 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634195 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634197 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634200 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634203 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634205 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634208 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634211 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634213 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634216 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634218 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:13.636364 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634221 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634223 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634226 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634228 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634231 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634233 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634236 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634238 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634241 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634244 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634247 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634250 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634252 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634255 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634258 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634260 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634263 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634265 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634268 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634271 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:13.636847 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634274 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634276 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634279 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634281 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634284 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634286 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634289 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634291 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634293 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634296 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634298 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634301 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634303 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634306 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634309 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634311 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634314 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634318 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634322 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:13.637390 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634325 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634328 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634330 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634335 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634338 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634340 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634343 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634346 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634348 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634351 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634353 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634356 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634359 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634361 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634364 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634366 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634368 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634371 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634374 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634376 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:13.637850 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634379 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634381 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634384 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634387 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634389 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634392 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634395 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634398 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634402 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634405 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634409 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634412 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634416 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634418 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634422 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634426 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634429 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634431 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634434 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:13.638331 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634437 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634439 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634442 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634444 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634447 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634449 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.634455 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634520 2558 flags.go:64] FLAG: --address="0.0.0.0" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634526 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634533 2558 flags.go:64] FLAG: --anonymous-auth="true" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634537 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634541 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634544 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634548 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634552 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634555 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634558 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634562 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634565 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634568 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634571 2558 flags.go:64] FLAG: --cgroup-root="" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634574 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634577 2558 flags.go:64] FLAG: --client-ca-file="" Jul 31 14:12:13.638829 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634580 2558 flags.go:64] FLAG: --cloud-provider="external" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634583 2558 flags.go:64] FLAG: --cluster-dns="[]" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634587 2558 flags.go:64] FLAG: --cluster-domain="" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634590 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634593 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634599 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634602 2558 flags.go:64] FLAG: --container-log-max-files="5" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634606 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634609 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634612 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634615 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634618 2558 flags.go:64] FLAG: --contention-profiling="false" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634621 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634625 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634628 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634630 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634635 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634638 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634641 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634644 2558 flags.go:64] FLAG: --enable-load-reader="false" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634647 2558 flags.go:64] FLAG: --enable-server="true" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634649 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634654 2558 flags.go:64] FLAG: --event-burst="100" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634657 2558 flags.go:64] FLAG: --event-qps="50" Jul 31 14:12:13.639399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634660 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634663 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634665 2558 flags.go:64] FLAG: --eviction-hard="" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634670 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634673 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634676 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634679 2558 flags.go:64] FLAG: --eviction-soft="" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634682 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634685 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634688 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634691 2558 flags.go:64] FLAG: --experimental-mounter-path="" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634694 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634696 2558 flags.go:64] FLAG: --fail-swap-on="true" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634700 2558 flags.go:64] FLAG: --feature-gates="" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634704 2558 flags.go:64] FLAG: --file-check-frequency="20s" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634707 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634710 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634713 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634716 2558 flags.go:64] FLAG: --healthz-port="10248" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634718 2558 flags.go:64] FLAG: --help="false" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634721 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-130-33.ec2.internal" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634724 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634727 2558 flags.go:64] FLAG: --http-check-frequency="20s" Jul 31 14:12:13.639967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634730 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634733 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634737 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634740 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634742 2558 flags.go:64] FLAG: --image-service-endpoint="" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634745 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634748 2558 flags.go:64] FLAG: --kube-api-burst="100" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634751 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634754 2558 flags.go:64] FLAG: --kube-api-qps="50" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634756 2558 flags.go:64] FLAG: --kube-reserved="" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634759 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634762 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634766 2558 flags.go:64] FLAG: --kubelet-cgroups="" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634769 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634772 2558 flags.go:64] FLAG: --lock-file="" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634774 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634777 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634781 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634786 2558 flags.go:64] FLAG: --log-json-split-stream="false" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634789 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634792 2558 flags.go:64] FLAG: --log-text-split-stream="false" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634795 2558 flags.go:64] FLAG: --logging-format="text" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634799 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634802 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 31 14:12:13.640564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634805 2558 flags.go:64] FLAG: --manifest-url="" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634808 2558 flags.go:64] FLAG: --manifest-url-header="" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634812 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634815 2558 flags.go:64] FLAG: --max-open-files="1000000" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634819 2558 flags.go:64] FLAG: --max-pods="110" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634822 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634825 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634828 2558 flags.go:64] FLAG: --memory-manager-policy="None" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634831 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634834 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634837 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634840 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634847 2558 flags.go:64] FLAG: --node-status-max-images="50" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634850 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634853 2558 flags.go:64] FLAG: --oom-score-adj="-999" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634856 2558 flags.go:64] FLAG: --pod-cidr="" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634858 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634864 2558 flags.go:64] FLAG: --pod-manifest-path="" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634867 2558 flags.go:64] FLAG: --pod-max-pids="-1" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634870 2558 flags.go:64] FLAG: --pods-per-core="0" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634873 2558 flags.go:64] FLAG: --port="10250" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634877 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634880 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05fff8b90d1ed7430" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634883 2558 flags.go:64] FLAG: --qos-reserved="" Jul 31 14:12:13.641301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634887 2558 flags.go:64] FLAG: --read-only-port="10255" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634889 2558 flags.go:64] FLAG: --register-node="true" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634892 2558 flags.go:64] FLAG: --register-with-taints="" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634896 2558 flags.go:64] FLAG: --registry-burst="10" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634899 2558 flags.go:64] FLAG: --registry-qps="5" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634901 2558 flags.go:64] FLAG: --reserved-cpus="" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634904 2558 flags.go:64] FLAG: --reserved-memory="" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634911 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634914 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634917 2558 flags.go:64] FLAG: --rotate-certificates="false" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634920 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634923 2558 flags.go:64] FLAG: --runonce="false" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634926 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634932 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634935 2558 flags.go:64] FLAG: --seccomp-default="false" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634938 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634941 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634944 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634947 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634950 2558 flags.go:64] FLAG: --storage-driver-password="root" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634953 2558 flags.go:64] FLAG: --storage-driver-secure="false" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634956 2558 flags.go:64] FLAG: --storage-driver-table="stats" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634960 2558 flags.go:64] FLAG: --storage-driver-user="root" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634962 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634965 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 31 14:12:13.642331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634968 2558 flags.go:64] FLAG: --system-cgroups="" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634971 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634976 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634981 2558 flags.go:64] FLAG: --tls-cert-file="" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634983 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634987 2558 flags.go:64] FLAG: --tls-min-version="" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634990 2558 flags.go:64] FLAG: --tls-private-key-file="" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634993 2558 flags.go:64] FLAG: --topology-manager-policy="none" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634995 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.634998 2558 flags.go:64] FLAG: --topology-manager-scope="container" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.635001 2558 flags.go:64] FLAG: --v="2" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.635019 2558 flags.go:64] FLAG: --version="false" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.635023 2558 flags.go:64] FLAG: --vmodule="" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.635026 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.635031 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635143 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635146 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635150 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635153 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635155 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635158 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635161 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635164 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:13.643441 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635167 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635170 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635172 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635175 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635178 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635181 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635184 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635186 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635189 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635191 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635194 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635196 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635200 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635203 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635205 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635208 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635211 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635213 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635216 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635218 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:13.644243 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635221 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635223 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635226 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635230 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635232 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635235 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635237 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635240 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635242 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635245 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635248 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635250 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635253 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635255 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635258 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635261 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635263 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635266 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635269 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635271 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:13.644729 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635274 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635276 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635278 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635281 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635285 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635287 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635290 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635293 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635295 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635298 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635300 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635303 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635307 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635310 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635313 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635318 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635321 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635324 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635327 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635329 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:13.645382 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635332 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635336 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635339 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635342 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635345 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635347 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635350 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635352 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635355 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635358 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635360 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635363 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635366 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635369 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635371 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635374 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635379 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635383 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635385 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:13.646112 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635388 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635390 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635393 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635395 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635398 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635401 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635403 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635406 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635410 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635412 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635415 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.635418 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.636068 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.643900 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.643923 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 31 14:12:13.646940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.645572 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.645589 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645688 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645695 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645701 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645706 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645710 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645714 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645719 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645723 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645727 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645731 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645736 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645740 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645744 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645749 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645753 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645757 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645761 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:13.647458 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645765 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645770 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645774 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645778 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645782 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645786 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645791 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645795 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645799 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645803 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645807 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645811 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645825 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645830 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645834 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645838 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645842 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645847 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645851 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:13.648024 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645855 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645859 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645863 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645867 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645871 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645876 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645880 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645884 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645890 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645897 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645902 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645907 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645911 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645916 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645920 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645924 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645929 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645933 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645938 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:13.648789 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645942 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645946 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645951 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645955 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645959 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645963 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645970 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645984 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645989 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.645994 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646000 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646021 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646026 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646031 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646037 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646042 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646047 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646051 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646056 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:13.649323 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646061 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646065 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646070 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646074 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646079 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646083 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646088 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646092 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646097 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646101 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646105 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646109 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646113 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646117 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646121 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646125 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646129 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646132 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646137 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646141 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:13.649782 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646146 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646158 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646163 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646168 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646172 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.646180 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646416 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646425 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646430 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646434 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646438 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646443 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646447 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646452 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646456 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 31 14:12:13.650292 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646460 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646464 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646468 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646473 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646477 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646481 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646485 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646497 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646502 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646506 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646510 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646514 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646518 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646523 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646527 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646531 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646536 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646541 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646545 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646556 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 31 14:12:13.650675 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646561 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646565 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646570 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646576 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646586 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646591 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646596 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646600 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646605 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646609 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646614 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646618 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646622 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646628 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646634 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646639 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646643 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646647 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646652 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 31 14:12:13.651189 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646656 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646661 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646666 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646670 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646674 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646678 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646682 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646686 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646690 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646695 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646699 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646704 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646708 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646719 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646724 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646728 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646732 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646736 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646740 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646744 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 31 14:12:13.651647 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646748 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646752 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646757 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646761 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646765 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646769 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646773 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646777 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646782 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646786 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646790 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646794 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646798 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646803 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646807 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646811 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646815 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646819 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646823 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646827 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 31 14:12:13.652138 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646831 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646837 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646842 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646847 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646851 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646855 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646866 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646871 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646876 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646880 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:13.646884 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.646891 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.647076 2558 server.go:962] "Client rotation is on, will bootstrap in background" Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.650907 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 31 14:12:13.652617 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.652214 2558 server.go:1019] "Starting client certificate rotation" Jul 31 14:12:13.652960 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.652311 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 14:12:13.652960 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.652343 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 31 14:12:13.654804 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.654789 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 14:12:13.658636 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.658617 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 31 14:12:13.670912 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.670897 2558 log.go:25] "Validated CRI v1 runtime API" Jul 31 14:12:13.674202 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.674189 2558 log.go:25] "Validated CRI v1 image API" Jul 31 14:12:13.675125 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.675110 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 31 14:12:13.679765 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.679741 2558 fs.go:135] Filesystem UUIDs: map[17ba988e-1af8-4a56-b230-e93da12551bb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ad92efd6-ae2c-4688-b00e-af7eeaf46730:/dev/nvme0n1p3] Jul 31 14:12:13.679855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.679766 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 31 14:12:13.685475 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.685371 2558 manager.go:216] Machine: {Timestamp:2026-07-31 14:12:13.68412923 +0000 UTC m=+0.324406898 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096934 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2694479a754c7ff85d380e4731615d SystemUUID:ec269447-9a75-4c7f-f85d-380e4731615d BootID:fc6da98e-6130-4456-8ae5-9c34cd418372 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:ce:a2:e2:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:ce:a2:e2:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:a1:8c:4f:6f:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 31 14:12:13.685475 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.685471 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 31 14:12:13.685603 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.685548 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 31 14:12:13.686574 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.686553 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 14:12:13.688890 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.688862 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 31 14:12:13.689036 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.688893 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-33.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 31 14:12:13.690060 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.690049 2558 topology_manager.go:138] "Creating topology manager with none policy" Jul 31 14:12:13.690094 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.690062 2558 container_manager_linux.go:309] "Creating device plugin manager" Jul 31 14:12:13.690094 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.690069 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 14:12:13.690662 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.690653 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 31 14:12:13.691639 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.691629 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 31 14:12:13.695695 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.695682 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 31 14:12:13.695812 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.695803 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 31 14:12:13.695861 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.695853 2558 kubelet.go:486] "Attempting to sync node with API server" Jul 31 14:12:13.695889 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.695872 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 31 14:12:13.695889 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.695883 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 31 14:12:13.695944 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.695894 2558 kubelet.go:398] "Adding apiserver pod source" Jul 31 14:12:13.695944 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.695930 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 31 14:12:13.696475 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.696445 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 14:12:13.696598 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.696525 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 31 14:12:13.700345 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.700324 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 31 14:12:13.701919 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.701902 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 31 14:12:13.701994 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.701934 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 31 14:12:13.703407 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703395 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 31 14:12:13.703454 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703411 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 31 14:12:13.703454 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703418 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 31 14:12:13.703454 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703426 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 31 14:12:13.703454 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703439 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 31 14:12:13.703454 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703448 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 31 14:12:13.703454 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703456 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 31 14:12:13.703607 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703461 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 31 14:12:13.703607 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703468 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 31 14:12:13.703607 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703475 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 31 14:12:13.703607 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703483 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 31 14:12:13.703607 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.703488 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 31 14:12:13.704545 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.704534 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 31 14:12:13.704584 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.704547 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 31 14:12:13.706824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.706803 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-33.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 31 14:12:13.706910 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.706883 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-33.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 31 14:12:13.706967 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.706894 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 31 14:12:13.707864 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.707852 2558 server.go:1268] "Started kubelet" Jul 31 14:12:13.707991 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.707932 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 31 14:12:13.708094 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.707964 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 31 14:12:13.708094 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.708034 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 31 14:12:13.708678 ip-10-0-130-33 systemd[1]: Started Kubernetes Kubelet. Jul 31 14:12:13.709488 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.709468 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 31 14:12:13.710371 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.710356 2558 server.go:310] "Adding debug handlers to kubelet server" Jul 31 14:12:13.714168 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.713308 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-33.ec2.internal.18c765473094a898 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-33.ec2.internal,UID:ip-10-0-130-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-33.ec2.internal,},FirstTimestamp:2026-07-31 14:12:13.707823256 +0000 UTC m=+0.348100924,LastTimestamp:2026-07-31 14:12:13.707823256 +0000 UTC m=+0.348100924,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-33.ec2.internal,}" Jul 31 14:12:13.714459 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.714430 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 31 14:12:13.717649 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.717624 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 31 14:12:13.717728 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.717634 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 31 14:12:13.717728 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.717706 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:13.718422 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718403 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 31 14:12:13.718518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718400 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 31 14:12:13.718518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718439 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 31 14:12:13.718615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718526 2558 factory.go:55] Registering systemd factory Jul 31 14:12:13.718615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718542 2558 factory.go:223] Registration of the systemd container factory successfully Jul 31 14:12:13.718615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718563 2558 reconstruct.go:97] "Volume reconstruction finished" Jul 31 14:12:13.718615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718569 2558 reconciler.go:29] "Reconciler: start to sync state" Jul 31 14:12:13.718778 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718767 2558 factory.go:153] Registering CRI-O factory Jul 31 14:12:13.718825 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718779 2558 factory.go:223] Registration of the crio container factory successfully Jul 31 14:12:13.718875 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718832 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 31 14:12:13.718875 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718862 2558 factory.go:103] Registering Raw factory Jul 31 14:12:13.718875 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.718873 2558 manager.go:1196] Started watching for new ooms in manager Jul 31 14:12:13.719269 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.719256 2558 manager.go:349] Starting recovery of all containers Jul 31 14:12:13.728681 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.728648 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sjrc5" Jul 31 14:12:13.730279 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.730155 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 31 14:12:13.730279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.730188 2558 manager.go:354] Recovery completed Jul 31 14:12:13.730279 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.730265 2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-33.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 31 14:12:13.734989 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.734970 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sjrc5" Jul 31 14:12:13.737057 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.737044 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:13.739183 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.739165 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:13.739266 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.739196 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:13.739266 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.739211 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:13.739881 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.739861 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 31 14:12:13.739881 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.739880 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 31 14:12:13.740051 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.739868 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:13.740051 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.739899 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 31 14:12:13.740614 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.740600 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:13.740680 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.740624 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:13.740680 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.740633 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:13.741134 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.741121 2558 policy_none.go:49] "None policy: Start" Jul 31 14:12:13.741181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.741139 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 31 14:12:13.741181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.741149 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 31 14:12:13.742130 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.742118 2558 policy_none.go:47] "Start" Jul 31 14:12:13.765618 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.765476 2558 manager.go:337] "Starting Device Plugin manager" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.765629 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.765642 2558 server.go:87] "Starting device plugin registration server" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.765873 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.765884 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.766022 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.766104 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.766115 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.766591 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 31 14:12:13.782735 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.766633 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:13.843653 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.843612 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 31 14:12:13.844781 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.844762 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 31 14:12:13.844841 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.844807 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 31 14:12:13.844841 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.844831 2558 kubelet.go:2448] "Starting kubelet main sync loop" Jul 31 14:12:13.844935 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.844868 2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 31 14:12:13.846669 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.846651 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:13.866567 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.866555 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:13.868147 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.868132 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:13.868221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.868159 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:13.868221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.868169 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:13.868221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.868190 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-33.ec2.internal" Jul 31 14:12:13.877113 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.877048 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-33.ec2.internal" Jul 31 14:12:13.877113 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.877071 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-33.ec2.internal\": node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:13.902740 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.902722 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:13.945570 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.945548 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal"] Jul 31 14:12:13.945651 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.945611 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:13.946412 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.946398 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:13.946528 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.946422 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:13.946528 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.946432 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:13.947660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.947644 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:13.947785 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.947772 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:13.947840 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.947803 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:13.948335 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.948319 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:13.948335 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.948333 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:13.948459 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.948342 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:13.948459 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.948354 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:13.948459 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.948354 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:13.948459 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.948449 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:13.949425 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.949410 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" Jul 31 14:12:13.949482 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.949450 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 31 14:12:13.950435 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.950419 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientMemory" Jul 31 14:12:13.950503 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.950445 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasNoDiskPressure" Jul 31 14:12:13.950503 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:13.950455 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeHasSufficientPID" Jul 31 14:12:13.978631 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.978613 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-33.ec2.internal\" not found" node="ip-10-0-130-33.ec2.internal" Jul 31 14:12:13.981852 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:13.981837 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-33.ec2.internal\" not found" node="ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.003290 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:14.003274 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:14.103675 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:14.103635 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:14.119964 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.119943 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8f44f24510d413518febbf8df137749-config\") pod \"kube-apiserver-proxy-ip-10-0-130-33.ec2.internal\" (UID: \"e8f44f24510d413518febbf8df137749\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.120048 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.119975 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90f53f212e24b1ebe548b639469c75cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal\" (UID: \"90f53f212e24b1ebe548b639469c75cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.120048 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.119998 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f53f212e24b1ebe548b639469c75cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal\" (UID: \"90f53f212e24b1ebe548b639469c75cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.204388 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:14.204324 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:14.220679 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.220657 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90f53f212e24b1ebe548b639469c75cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal\" (UID: \"90f53f212e24b1ebe548b639469c75cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.220736 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.220688 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f53f212e24b1ebe548b639469c75cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal\" (UID: \"90f53f212e24b1ebe548b639469c75cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.220736 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.220708 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8f44f24510d413518febbf8df137749-config\") pod \"kube-apiserver-proxy-ip-10-0-130-33.ec2.internal\" (UID: \"e8f44f24510d413518febbf8df137749\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.220800 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.220753 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f53f212e24b1ebe548b639469c75cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal\" (UID: \"90f53f212e24b1ebe548b639469c75cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.220800 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.220774 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8f44f24510d413518febbf8df137749-config\") pod \"kube-apiserver-proxy-ip-10-0-130-33.ec2.internal\" (UID: \"e8f44f24510d413518febbf8df137749\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.220800 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.220753 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90f53f212e24b1ebe548b639469c75cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal\" (UID: \"90f53f212e24b1ebe548b639469c75cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.282850 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.282810 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.284965 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.284946 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" Jul 31 14:12:14.304597 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:14.304578 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:14.405046 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:14.404995 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:14.505584 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:14.505511 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:14.606019 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:14.605976 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:14.651491 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.651468 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 31 14:12:14.651932 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:14.651643 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 31 14:12:14.707020 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:14.706985 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-33.ec2.internal\" not found" Jul 31 14:12:15.873196 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.873166 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:07:14 +0000 UTC" deadline="2028-04-21 14:27:04.642050742 +0000 UTC" Jul 31 14:12:15.873196 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.873194 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15120h14m48.768860316s" Jul 31 14:12:15.880855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.880839 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:15.949971 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.949951 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" Jul 31 14:12:15.953620 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:15.953594 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f44f24510d413518febbf8df137749.slice/crio-06ce920634cf9e2b1baeb20b47122c1b64b92b3461a6fbf365d7b7c99a6937de WatchSource:0}: Error finding container 06ce920634cf9e2b1baeb20b47122c1b64b92b3461a6fbf365d7b7c99a6937de: Status 404 returned error can't find the container with id 06ce920634cf9e2b1baeb20b47122c1b64b92b3461a6fbf365d7b7c99a6937de Jul 31 14:12:15.953826 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:15.953803 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f53f212e24b1ebe548b639469c75cc.slice/crio-36c551753142ec19fffb5737e20d0c0388fd6b1adf24321e5d2083d40afc413e WatchSource:0}: Error finding container 36c551753142ec19fffb5737e20d0c0388fd6b1adf24321e5d2083d40afc413e: Status 404 returned error can't find the container with id 36c551753142ec19fffb5737e20d0c0388fd6b1adf24321e5d2083d40afc413e Jul 31 14:12:15.958602 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.958589 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:12:15.963081 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.963064 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 14:12:15.964810 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.964794 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" Jul 31 14:12:15.967489 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.967472 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:15.973272 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.973257 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 31 14:12:15.980297 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.980243 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" event={"ID":"90f53f212e24b1ebe548b639469c75cc","Type":"ContainerStarted","Data":"36c551753142ec19fffb5737e20d0c0388fd6b1adf24321e5d2083d40afc413e"} Jul 31 14:12:15.981267 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:15.981250 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" event={"ID":"e8f44f24510d413518febbf8df137749","Type":"ContainerStarted","Data":"06ce920634cf9e2b1baeb20b47122c1b64b92b3461a6fbf365d7b7c99a6937de"} Jul 31 14:12:16.326346 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.326263 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:16.591453 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.591306 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 31 14:12:16.828781 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.828750 2558 apiserver.go:52] "Watching apiserver" Jul 31 14:12:16.838334 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.838303 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 31 14:12:16.840780 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.840751 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-kdtjx","openshift-network-diagnostics/network-check-target-x2qjh","kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn","openshift-image-registry/node-ca-kd9wm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal","openshift-network-operator/iptables-alerter-dzxbx","openshift-ovn-kubernetes/ovnkube-node-z9hbr","kube-system/global-pull-secret-syncer-vlwj7","kube-system/konnectivity-agent-5t4bh","openshift-cluster-node-tuning-operator/tuned-f8tr8","openshift-dns/node-resolver-8wjhs","openshift-multus/multus-4gd4d","openshift-multus/multus-additional-cni-plugins-b8tpw"] Jul 31 14:12:16.842429 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.842365 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:16.842495 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.842443 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:16.843497 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.843478 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:16.843579 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.843554 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:16.844577 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.844555 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.845809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.845792 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.846803 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.846781 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 31 14:12:16.847439 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.847081 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.847439 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.847329 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fg966\"" Jul 31 14:12:16.847616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.847606 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 31 14:12:16.847616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.847610 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 31 14:12:16.848129 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.848110 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 31 14:12:16.848363 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.848340 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 31 14:12:16.848517 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.848501 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.848582 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.848565 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 31 14:12:16.848675 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.848653 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxpfb\"" Jul 31 14:12:16.849809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.849788 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jt8kf\"" Jul 31 14:12:16.849809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.849801 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.849975 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.849819 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:12:16.849975 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.849858 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:16.849975 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.849944 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 31 14:12:16.850426 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.850398 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 31 14:12:16.850659 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.850642 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 31 14:12:16.851020 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.850979 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:16.851397 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.851377 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-646fl\"" Jul 31 14:12:16.851886 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.851865 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 31 14:12:16.852003 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.851899 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 31 14:12:16.852680 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.852247 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.852878 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.852861 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 31 14:12:16.853488 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.853361 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 31 14:12:16.853488 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.853378 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 31 14:12:16.853488 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.853427 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 31 14:12:16.853704 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.853683 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:16.853924 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.853856 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2srmt\"" Jul 31 14:12:16.854140 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.854114 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 31 14:12:16.854279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.854211 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 31 14:12:16.854428 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.854409 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:12:16.854885 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.854864 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 31 14:12:16.855048 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.854893 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rkgr\"" Jul 31 14:12:16.855213 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.855183 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.856395 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.856377 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 31 14:12:16.856478 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.856386 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kv7hd\"" Jul 31 14:12:16.857967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.856973 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.857967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.857698 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.858524 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sr4l2\"" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.858595 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.858674 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.858752 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.858761 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859078 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-systemd\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859119 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-var-lib-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859150 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859174 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-log-socket\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859206 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859235 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-env-overrides\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859263 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859291 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rzc\" (UniqueName: \"kubernetes.io/projected/4c0cf086-f032-4b04-adcb-23df0705d67c-kube-api-access-22rzc\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859318 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-etc-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859348 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-ovn\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859377 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-cni-bin\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859459 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b6b34e3-8489-4f7e-aee5-df61d5cf87cc-konnectivity-ca\") pod \"konnectivity-agent-5t4bh\" (UID: \"9b6b34e3-8489-4f7e-aee5-df61d5cf87cc\") " pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859506 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5ea2efc-9e78-45d5-a382-7233eab15fce-host-slash\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.859968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859549 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-node-log\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859580 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-cni-netd\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859616 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8ct\" (UniqueName: \"kubernetes.io/projected/b8dd22be-cc3b-43b6-9dcd-61faa79da219-kube-api-access-4h8ct\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859664 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b6b34e3-8489-4f7e-aee5-df61d5cf87cc-agent-certs\") pod \"konnectivity-agent-5t4bh\" (UID: \"9b6b34e3-8489-4f7e-aee5-df61d5cf87cc\") " pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859695 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4z78\" (UniqueName: \"kubernetes.io/projected/26920aff-943f-4177-8777-d186d5e99fc3-kube-api-access-f4z78\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859742 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-systemd-units\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859769 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-slash\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859794 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859839 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859873 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c0cf086-f032-4b04-adcb-23df0705d67c-serviceca\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859932 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovn-node-metrics-cert\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859966 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.859973 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce208340-fc19-4886-b1ce-1e4d7807dbe2-kubelet-config\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860038 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce208340-fc19-4886-b1ce-1e4d7807dbe2-dbus\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860124 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860163 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5ea2efc-9e78-45d5-a382-7233eab15fce-iptables-alerter-script\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860202 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovnkube-config\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.860834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860216 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860234 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860310 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860345 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnrrn\" (UniqueName: \"kubernetes.io/projected/2f51637b-25e7-4268-a752-37b03d2a44d9-kube-api-access-pnrrn\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860399 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-kubelet\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860421 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fk7r5\"" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860451 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-run-netns\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860522 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860656 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860715 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860745 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0cf086-f032-4b04-adcb-23df0705d67c-host\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860801 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovnkube-script-lib\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860829 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-device-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.861624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.860877 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7t59\" (UniqueName: \"kubernetes.io/projected/c5ea2efc-9e78-45d5-a382-7233eab15fce-kube-api-access-z7t59\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.863380 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.863213 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 31 14:12:16.951058 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.951030 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 31 14:12:16.961203 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961174 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-tuned\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.961303 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961219 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-node-log\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.961303 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961244 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-cni-netd\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.961303 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961269 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8ct\" (UniqueName: \"kubernetes.io/projected/b8dd22be-cc3b-43b6-9dcd-61faa79da219-kube-api-access-4h8ct\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.961303 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961293 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-node-log\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.961504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961293 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809b2748-9aad-4e40-864e-2252739a4f03-tmp-dir\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:16.961504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961352 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-system-cni-dir\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.961504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961362 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-cni-netd\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.961504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961377 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-os-release\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.961504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961406 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d57b476a-2d1a-4caf-be2c-ea4677ca3602-cni-binary-copy\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.961504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961432 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4z78\" (UniqueName: \"kubernetes.io/projected/26920aff-943f-4177-8777-d186d5e99fc3-kube-api-access-f4z78\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:16.961504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961455 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-systemd-units\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.961504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961498 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-cni-multus\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961512 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-systemd-units\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961571 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-kubelet\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961601 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-multus-certs\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961628 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961654 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c0cf086-f032-4b04-adcb-23df0705d67c-serviceca\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961678 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961706 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961731 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-lib-modules\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961753 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d45a40-0428-4335-b168-eaf9a86789ae-tmp\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961769 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961774 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxk8\" (UniqueName: \"kubernetes.io/projected/d57b476a-2d1a-4caf-be2c-ea4677ca3602-kube-api-access-vlxk8\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961801 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce208340-fc19-4886-b1ce-1e4d7807dbe2-dbus\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.961824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961817 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovnkube-config\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961835 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d24p9\" (UniqueName: \"kubernetes.io/projected/70381455-7d7d-4682-9ad3-0c19ed448194-kube-api-access-d24p9\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961853 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-systemd\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961868 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-var-lib-kubelet\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961888 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-cni-bin\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961929 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961953 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961970 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-kubelet\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961984 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-run-netns\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.961998 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962014 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-cni-binary-copy\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962018 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce208340-fc19-4886-b1ce-1e4d7807dbe2-dbus\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962029 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhcz\" (UniqueName: \"kubernetes.io/projected/c9d45a40-0428-4335-b168-eaf9a86789ae-kube-api-access-ckhcz\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962044 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-daemon-config\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962059 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962076 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysconfig\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962091 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-cnibin\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.962539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962106 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-socket-dir-parent\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962138 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4c0cf086-f032-4b04-adcb-23df0705d67c-serviceca\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962148 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-systemd\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962174 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-log-socket\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962190 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-env-overrides\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962204 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-os-release\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962219 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-kubernetes\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962235 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-etc-kubernetes\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962254 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962270 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-cni-bin\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962283 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-run\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-system-cni-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962319 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-socket-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962333 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b6b34e3-8489-4f7e-aee5-df61d5cf87cc-konnectivity-ca\") pod \"konnectivity-agent-5t4bh\" (UID: \"9b6b34e3-8489-4f7e-aee5-df61d5cf87cc\") " pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962380 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-sys-fs\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962411 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5ea2efc-9e78-45d5-a382-7233eab15fce-host-slash\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962441 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-host\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.963468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962468 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b6b34e3-8489-4f7e-aee5-df61d5cf87cc-agent-certs\") pod \"konnectivity-agent-5t4bh\" (UID: \"9b6b34e3-8489-4f7e-aee5-df61d5cf87cc\") " pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962492 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-slash\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962519 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-modprobe-d\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962545 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962568 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovn-node-metrics-cert\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962576 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovnkube-config\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962593 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-cnibin\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962620 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce208340-fc19-4886-b1ce-1e4d7807dbe2-kubelet-config\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962645 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962671 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5ea2efc-9e78-45d5-a382-7233eab15fce-iptables-alerter-script\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962702 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/809b2748-9aad-4e40-864e-2252739a4f03-hosts-file\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.962738 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962730 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xcc\" (UniqueName: \"kubernetes.io/projected/809b2748-9aad-4e40-864e-2252739a4f03-kube-api-access-g8xcc\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962773 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysctl-d\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.962808 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:17.46279061 +0000 UTC m=+2.971169654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962827 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b6b34e3-8489-4f7e-aee5-df61d5cf87cc-konnectivity-ca\") pod \"konnectivity-agent-5t4bh\" (UID: \"9b6b34e3-8489-4f7e-aee5-df61d5cf87cc\") " pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962831 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-conf-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.964214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962846 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-kubelet\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962877 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnrrn\" (UniqueName: \"kubernetes.io/projected/2f51637b-25e7-4268-a752-37b03d2a44d9-kube-api-access-pnrrn\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962958 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-env-overrides\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962975 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce208340-fc19-4886-b1ce-1e4d7807dbe2-kubelet-config\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962075 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-registration-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963010 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.962980 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-run-netns\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963045 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-systemd\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963065 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-log-socket\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963121 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0cf086-f032-4b04-adcb-23df0705d67c-host\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963147 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovnkube-script-lib\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.963192 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963207 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-cni-bin\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.963226 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret podName:ce208340-fc19-4886-b1ce-1e4d7807dbe2 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:17.463215946 +0000 UTC m=+2.971594989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret") pod "global-pull-secret-syncer-vlwj7" (UID: "ce208340-fc19-4886-b1ce-1e4d7807dbe2") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963240 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5ea2efc-9e78-45d5-a382-7233eab15fce-host-slash\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963488 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5ea2efc-9e78-45d5-a382-7233eab15fce-iptables-alerter-script\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.964968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963525 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963645 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-slash\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963651 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovnkube-script-lib\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963729 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963769 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0cf086-f032-4b04-adcb-23df0705d67c-host\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963542 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysctl-conf\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963898 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-sys\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963958 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-k8s-cni-cncf-io\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.963994 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-hostroot\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964022 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-device-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964048 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7t59\" (UniqueName: \"kubernetes.io/projected/c5ea2efc-9e78-45d5-a382-7233eab15fce-kube-api-access-z7t59\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964176 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f51637b-25e7-4268-a752-37b03d2a44d9-device-dir\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964209 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-var-lib-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964254 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964268 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-var-lib-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964297 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964341 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.965634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964339 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:16.966358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964386 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.966358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964388 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-cni-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.966358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964414 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-netns\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:16.966358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964442 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rzc\" (UniqueName: \"kubernetes.io/projected/4c0cf086-f032-4b04-adcb-23df0705d67c-kube-api-access-22rzc\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:16.966358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964468 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-etc-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.966358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964505 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-ovn\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.966358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964537 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-etc-openvswitch\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.966358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.964620 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8dd22be-cc3b-43b6-9dcd-61faa79da219-run-ovn\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.967094 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.967072 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8dd22be-cc3b-43b6-9dcd-61faa79da219-ovn-node-metrics-cert\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.967194 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.967151 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b6b34e3-8489-4f7e-aee5-df61d5cf87cc-agent-certs\") pod \"konnectivity-agent-5t4bh\" (UID: \"9b6b34e3-8489-4f7e-aee5-df61d5cf87cc\") " pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:16.970808 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.970785 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8ct\" (UniqueName: \"kubernetes.io/projected/b8dd22be-cc3b-43b6-9dcd-61faa79da219-kube-api-access-4h8ct\") pod \"ovnkube-node-z9hbr\" (UID: \"b8dd22be-cc3b-43b6-9dcd-61faa79da219\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:16.971080 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.971031 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4z78\" (UniqueName: \"kubernetes.io/projected/26920aff-943f-4177-8777-d186d5e99fc3-kube-api-access-f4z78\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:16.972222 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.972196 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:16.972222 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.972220 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:16.972429 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.972233 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2664k for pod openshift-network-diagnostics/network-check-target-x2qjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:16.972429 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:16.972297 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k podName:a72ae63b-a0bd-42de-8108-45c128b023ed nodeName:}" failed. No retries permitted until 2026-07-31 14:12:17.472280742 +0000 UTC m=+2.980659805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2664k" (UniqueName: "kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k") pod "network-check-target-x2qjh" (UID: "a72ae63b-a0bd-42de-8108-45c128b023ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:16.973399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.973377 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7t59\" (UniqueName: \"kubernetes.io/projected/c5ea2efc-9e78-45d5-a382-7233eab15fce-kube-api-access-z7t59\") pod \"iptables-alerter-dzxbx\" (UID: \"c5ea2efc-9e78-45d5-a382-7233eab15fce\") " pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:16.974014 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.973997 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnrrn\" (UniqueName: \"kubernetes.io/projected/2f51637b-25e7-4268-a752-37b03d2a44d9-kube-api-access-pnrrn\") pod \"aws-ebs-csi-driver-node-xp9cn\" (UID: \"2f51637b-25e7-4268-a752-37b03d2a44d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:16.983853 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.983834 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m4wql" Jul 31 14:12:16.991094 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:16.991077 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rzc\" (UniqueName: \"kubernetes.io/projected/4c0cf086-f032-4b04-adcb-23df0705d67c-kube-api-access-22rzc\") pod \"node-ca-kd9wm\" (UID: \"4c0cf086-f032-4b04-adcb-23df0705d67c\") " pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:17.004400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.004376 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m4wql" Jul 31 14:12:17.064754 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064726 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-sys\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.064754 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064758 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-k8s-cni-cncf-io\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.064981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064778 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-hostroot\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.064981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064837 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-k8s-cni-cncf-io\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.064981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064865 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-sys\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.064981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064888 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-hostroot\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.064981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064926 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.064981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064956 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-cni-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.064981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.064980 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-netns\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065007 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-tuned\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065035 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809b2748-9aad-4e40-864e-2252739a4f03-tmp-dir\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065058 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-system-cni-dir\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065065 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-netns\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065035 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-cni-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065081 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-os-release\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065118 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d57b476a-2d1a-4caf-be2c-ea4677ca3602-cni-binary-copy\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065146 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-cni-multus\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065179 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-kubelet\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065200 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-multus-certs\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065224 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065247 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065273 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-lib-modules\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065294 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d45a40-0428-4335-b168-eaf9a86789ae-tmp\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065311 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-kubelet\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065441 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-lib-modules\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065533 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065584 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-cni-multus\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065589 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-os-release\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065315 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxk8\" (UniqueName: \"kubernetes.io/projected/d57b476a-2d1a-4caf-be2c-ea4677ca3602-kube-api-access-vlxk8\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065640 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d24p9\" (UniqueName: \"kubernetes.io/projected/70381455-7d7d-4682-9ad3-0c19ed448194-kube-api-access-d24p9\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065667 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-systemd\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065691 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-var-lib-kubelet\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.065757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065713 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-cni-bin\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065745 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-cni-binary-copy\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065800 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhcz\" (UniqueName: \"kubernetes.io/projected/c9d45a40-0428-4335-b168-eaf9a86789ae-kube-api-access-ckhcz\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065805 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065823 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-daemon-config\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065850 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysconfig\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065862 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-run-multus-certs\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065874 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-cnibin\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065899 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-socket-dir-parent\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065973 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-system-cni-dir\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.065981 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-socket-dir-parent\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066009 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066017 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-os-release\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066036 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-systemd\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066046 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-kubernetes\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066072 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-etc-kubernetes\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066085 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-var-lib-kubelet\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.066388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066103 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d57b476a-2d1a-4caf-be2c-ea4677ca3602-cni-binary-copy\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066110 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-run\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066128 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-host-var-lib-cni-bin\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066156 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-run\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066162 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-system-cni-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066191 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-host\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066213 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-os-release\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066218 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-modprobe-d\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066268 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-etc-kubernetes\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066311 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-host\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066343 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-modprobe-d\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066356 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-kubernetes\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066389 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-cnibin\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066398 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-system-cni-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066435 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70381455-7d7d-4682-9ad3-0c19ed448194-cnibin\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066437 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/809b2748-9aad-4e40-864e-2252739a4f03-hosts-file\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066472 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xcc\" (UniqueName: \"kubernetes.io/projected/809b2748-9aad-4e40-864e-2252739a4f03-kube-api-access-g8xcc\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066480 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/809b2748-9aad-4e40-864e-2252739a4f03-hosts-file\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:17.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066499 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysctl-d\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066524 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-conf-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066534 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70381455-7d7d-4682-9ad3-0c19ed448194-cni-binary-copy\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066553 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysctl-conf\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066645 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysconfig\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066668 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysctl-d\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066473 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-daemon-config\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066687 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-multus-conf-dir\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066697 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-sysctl-conf\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066735 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d57b476a-2d1a-4caf-be2c-ea4677ca3602-cnibin\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.066814 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809b2748-9aad-4e40-864e-2252739a4f03-tmp-dir\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.067459 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9d45a40-0428-4335-b168-eaf9a86789ae-etc-tuned\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.068033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.067751 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d45a40-0428-4335-b168-eaf9a86789ae-tmp\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.072841 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.072789 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxk8\" (UniqueName: \"kubernetes.io/projected/d57b476a-2d1a-4caf-be2c-ea4677ca3602-kube-api-access-vlxk8\") pod \"multus-4gd4d\" (UID: \"d57b476a-2d1a-4caf-be2c-ea4677ca3602\") " pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.073884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.073861 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhcz\" (UniqueName: \"kubernetes.io/projected/c9d45a40-0428-4335-b168-eaf9a86789ae-kube-api-access-ckhcz\") pod \"tuned-f8tr8\" (UID: \"c9d45a40-0428-4335-b168-eaf9a86789ae\") " pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.074109 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.074089 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xcc\" (UniqueName: \"kubernetes.io/projected/809b2748-9aad-4e40-864e-2252739a4f03-kube-api-access-g8xcc\") pod \"node-resolver-8wjhs\" (UID: \"809b2748-9aad-4e40-864e-2252739a4f03\") " pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:17.074494 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.074467 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d24p9\" (UniqueName: \"kubernetes.io/projected/70381455-7d7d-4682-9ad3-0c19ed448194-kube-api-access-d24p9\") pod \"multus-additional-cni-plugins-b8tpw\" (UID: \"70381455-7d7d-4682-9ad3-0c19ed448194\") " pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.156774 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.156704 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" Jul 31 14:12:17.165203 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.165177 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kd9wm" Jul 31 14:12:17.174817 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.174796 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" Jul 31 14:12:17.181986 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.181968 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dzxbx" Jul 31 14:12:17.189070 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.189050 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:17.196707 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.196688 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:17.203753 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.203737 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" Jul 31 14:12:17.210967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.210950 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8wjhs" Jul 31 14:12:17.215251 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.215234 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4gd4d" Jul 31 14:12:17.470872 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.470806 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:17.470872 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.470848 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:17.471088 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.470959 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:17.471088 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.470980 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:17.471088 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.471011 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret podName:ce208340-fc19-4886-b1ce-1e4d7807dbe2 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:18.470997256 +0000 UTC m=+3.979376299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret") pod "global-pull-secret-syncer-vlwj7" (UID: "ce208340-fc19-4886-b1ce-1e4d7807dbe2") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:17.471088 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.471040 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:18.47102224 +0000 UTC m=+3.979401299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:17.571962 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.571928 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:17.572137 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.572119 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:17.572179 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.572141 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:17.572179 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.572151 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2664k for pod openshift-network-diagnostics/network-check-target-x2qjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:17.572276 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.572203 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k podName:a72ae63b-a0bd-42de-8108-45c128b023ed nodeName:}" failed. No retries permitted until 2026-07-31 14:12:18.572187281 +0000 UTC m=+4.080566325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2664k" (UniqueName: "kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k") pod "network-check-target-x2qjh" (UID: "a72ae63b-a0bd-42de-8108-45c128b023ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:17.790539 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.790495 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809b2748_9aad_4e40_864e_2252739a4f03.slice/crio-bc993ee44cad415c61f1e2df22c29842cca87f89a1061818967b992f33c5d99e WatchSource:0}: Error finding container bc993ee44cad415c61f1e2df22c29842cca87f89a1061818967b992f33c5d99e: Status 404 returned error can't find the container with id bc993ee44cad415c61f1e2df22c29842cca87f89a1061818967b992f33c5d99e Jul 31 14:12:17.791287 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.791260 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ea2efc_9e78_45d5_a382_7233eab15fce.slice/crio-eef7204495cc4648f48604c43fcceaa83ab7a3e5cb2e8d03f3f66fd7a91648ed WatchSource:0}: Error finding container eef7204495cc4648f48604c43fcceaa83ab7a3e5cb2e8d03f3f66fd7a91648ed: Status 404 returned error can't find the container with id eef7204495cc4648f48604c43fcceaa83ab7a3e5cb2e8d03f3f66fd7a91648ed Jul 31 14:12:17.793471 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.793444 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57b476a_2d1a_4caf_be2c_ea4677ca3602.slice/crio-7a8ae75cd3e8482d7acccdda80050fe85b40bbb46fa3a5725aafa848704d4e9e WatchSource:0}: Error finding container 7a8ae75cd3e8482d7acccdda80050fe85b40bbb46fa3a5725aafa848704d4e9e: Status 404 returned error can't find the container with id 7a8ae75cd3e8482d7acccdda80050fe85b40bbb46fa3a5725aafa848704d4e9e Jul 31 14:12:17.796447 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.796397 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d45a40_0428_4335_b168_eaf9a86789ae.slice/crio-b27567621117b406ea3d3b57f8a7608cf2eda92b363e92f653539a6eaba861f1 WatchSource:0}: Error finding container b27567621117b406ea3d3b57f8a7608cf2eda92b363e92f653539a6eaba861f1: Status 404 returned error can't find the container with id b27567621117b406ea3d3b57f8a7608cf2eda92b363e92f653539a6eaba861f1 Jul 31 14:12:17.797828 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.797783 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0cf086_f032_4b04_adcb_23df0705d67c.slice/crio-19e1cf913630c128d60caf4937760570af35f25597ed9f3752e344878fce805f WatchSource:0}: Error finding container 19e1cf913630c128d60caf4937760570af35f25597ed9f3752e344878fce805f: Status 404 returned error can't find the container with id 19e1cf913630c128d60caf4937760570af35f25597ed9f3752e344878fce805f Jul 31 14:12:17.798692 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.798668 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dd22be_cc3b_43b6_9dcd_61faa79da219.slice/crio-889f020ec42351b6dde75347c1f5cb888554d406d7e737cbb98446643adeec7c WatchSource:0}: Error finding container 889f020ec42351b6dde75347c1f5cb888554d406d7e737cbb98446643adeec7c: Status 404 returned error can't find the container with id 889f020ec42351b6dde75347c1f5cb888554d406d7e737cbb98446643adeec7c Jul 31 14:12:17.799983 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.799963 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6b34e3_8489_4f7e_aee5_df61d5cf87cc.slice/crio-af1858b70f9f0783d4c7144cc4dac300d67fa3de7245933adbcb883452c94740 WatchSource:0}: Error finding container af1858b70f9f0783d4c7144cc4dac300d67fa3de7245933adbcb883452c94740: Status 404 returned error can't find the container with id af1858b70f9f0783d4c7144cc4dac300d67fa3de7245933adbcb883452c94740 Jul 31 14:12:17.801308 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.801046 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f51637b_25e7_4268_a752_37b03d2a44d9.slice/crio-9ba7b56fb214cc29095f17ba952c4e63d8186dfaead9f7ab48b917a29ccb7bcd WatchSource:0}: Error finding container 9ba7b56fb214cc29095f17ba952c4e63d8186dfaead9f7ab48b917a29ccb7bcd: Status 404 returned error can't find the container with id 9ba7b56fb214cc29095f17ba952c4e63d8186dfaead9f7ab48b917a29ccb7bcd Jul 31 14:12:17.802022 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:17.801990 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70381455_7d7d_4682_9ad3_0c19ed448194.slice/crio-dff773b97002c272715077204ef568772167b816776227f915bb76cfb94f02d8 WatchSource:0}: Error finding container dff773b97002c272715077204ef568772167b816776227f915bb76cfb94f02d8: Status 404 returned error can't find the container with id dff773b97002c272715077204ef568772167b816776227f915bb76cfb94f02d8 Jul 31 14:12:17.977230 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.977069 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:17.977778 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:17.977303 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:17.986051 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.986023 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" event={"ID":"e8f44f24510d413518febbf8df137749","Type":"ContainerStarted","Data":"73dc27c367e7670988eb783e0a8f35d9245037ad6287dbc1104821d1be4dc6ff"} Jul 31 14:12:17.987191 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.987169 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" event={"ID":"70381455-7d7d-4682-9ad3-0c19ed448194","Type":"ContainerStarted","Data":"dff773b97002c272715077204ef568772167b816776227f915bb76cfb94f02d8"} Jul 31 14:12:17.988087 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.988067 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" event={"ID":"2f51637b-25e7-4268-a752-37b03d2a44d9","Type":"ContainerStarted","Data":"9ba7b56fb214cc29095f17ba952c4e63d8186dfaead9f7ab48b917a29ccb7bcd"} Jul 31 14:12:17.989052 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.989033 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5t4bh" event={"ID":"9b6b34e3-8489-4f7e-aee5-df61d5cf87cc","Type":"ContainerStarted","Data":"af1858b70f9f0783d4c7144cc4dac300d67fa3de7245933adbcb883452c94740"} Jul 31 14:12:17.990123 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.990106 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"889f020ec42351b6dde75347c1f5cb888554d406d7e737cbb98446643adeec7c"} Jul 31 14:12:17.991102 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.991085 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kd9wm" event={"ID":"4c0cf086-f032-4b04-adcb-23df0705d67c","Type":"ContainerStarted","Data":"19e1cf913630c128d60caf4937760570af35f25597ed9f3752e344878fce805f"} Jul 31 14:12:17.992038 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.992020 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4gd4d" event={"ID":"d57b476a-2d1a-4caf-be2c-ea4677ca3602","Type":"ContainerStarted","Data":"7a8ae75cd3e8482d7acccdda80050fe85b40bbb46fa3a5725aafa848704d4e9e"} Jul 31 14:12:17.992880 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.992861 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" event={"ID":"c9d45a40-0428-4335-b168-eaf9a86789ae","Type":"ContainerStarted","Data":"b27567621117b406ea3d3b57f8a7608cf2eda92b363e92f653539a6eaba861f1"} Jul 31 14:12:17.993947 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.993919 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dzxbx" event={"ID":"c5ea2efc-9e78-45d5-a382-7233eab15fce","Type":"ContainerStarted","Data":"eef7204495cc4648f48604c43fcceaa83ab7a3e5cb2e8d03f3f66fd7a91648ed"} Jul 31 14:12:17.994858 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:17.994835 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8wjhs" event={"ID":"809b2748-9aad-4e40-864e-2252739a4f03","Type":"ContainerStarted","Data":"bc993ee44cad415c61f1e2df22c29842cca87f89a1061818967b992f33c5d99e"} Jul 31 14:12:18.007790 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:18.007752 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:07:16 +0000 UTC" deadline="2028-03-12 09:46:29.137092188 +0000 UTC" Jul 31 14:12:18.007790 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:18.007789 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14155h34m11.129306294s" Jul 31 14:12:18.484777 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:18.484689 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:18.484964 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:18.484783 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:18.485046 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.485024 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:18.485106 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.485091 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret podName:ce208340-fc19-4886-b1ce-1e4d7807dbe2 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:20.485072278 +0000 UTC m=+5.993451342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret") pod "global-pull-secret-syncer-vlwj7" (UID: "ce208340-fc19-4886-b1ce-1e4d7807dbe2") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:18.485593 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.485570 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:18.485696 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.485623 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:20.485609407 +0000 UTC m=+5.993988452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:18.585461 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:18.585424 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:18.585667 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.585608 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:18.585667 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.585630 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:18.585667 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.585643 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2664k for pod openshift-network-diagnostics/network-check-target-x2qjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:18.585837 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.585702 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k podName:a72ae63b-a0bd-42de-8108-45c128b023ed nodeName:}" failed. No retries permitted until 2026-07-31 14:12:20.585683849 +0000 UTC m=+6.094062896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2664k" (UniqueName: "kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k") pod "network-check-target-x2qjh" (UID: "a72ae63b-a0bd-42de-8108-45c128b023ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:18.978099 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:18.978024 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:18.978643 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.978218 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:18.978866 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:18.978713 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:18.978866 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:18.978827 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:19.008161 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:19.007417 2558 generic.go:356] "Generic (PLEG): container finished" podID="90f53f212e24b1ebe548b639469c75cc" containerID="5a58995c0a4562177926f47e7e9b72d19a4e8086e3bc7d269101a20dffefac0a" exitCode=0 Jul 31 14:12:19.008161 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:19.008092 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" event={"ID":"90f53f212e24b1ebe548b639469c75cc","Type":"ContainerDied","Data":"5a58995c0a4562177926f47e7e9b72d19a4e8086e3bc7d269101a20dffefac0a"} Jul 31 14:12:19.008401 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:19.008228 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:07:16 +0000 UTC" deadline="2028-03-26 02:00:53.988809559 +0000 UTC" Jul 31 14:12:19.008401 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:19.008249 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14483h48m34.980563269s" Jul 31 14:12:19.024695 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:19.024562 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-33.ec2.internal" podStartSLOduration=4.024542382 podStartE2EDuration="4.024542382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:17.99943988 +0000 UTC m=+3.507818945" watchObservedRunningTime="2026-07-31 14:12:19.024542382 +0000 UTC m=+4.532921449" Jul 31 14:12:19.977435 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:19.976985 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:19.977435 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:19.977149 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:20.022968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:20.022564 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" event={"ID":"90f53f212e24b1ebe548b639469c75cc","Type":"ContainerStarted","Data":"42dbfc2e239c65879169e213c4f5624c5e9a623d557a33d5d0a17574db742aed"} Jul 31 14:12:20.501743 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:20.501708 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:20.501938 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:20.501780 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:20.502017 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.501938 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:20.502017 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.502006 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret podName:ce208340-fc19-4886-b1ce-1e4d7807dbe2 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:24.501986351 +0000 UTC m=+10.010365402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret") pod "global-pull-secret-syncer-vlwj7" (UID: "ce208340-fc19-4886-b1ce-1e4d7807dbe2") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:20.502435 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.502417 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:20.502519 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.502474 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:24.502458201 +0000 UTC m=+10.010837246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:20.602955 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:20.602917 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:20.603142 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.603055 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:20.603142 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.603086 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:20.603142 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.603100 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2664k for pod openshift-network-diagnostics/network-check-target-x2qjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:20.603304 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.603162 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k podName:a72ae63b-a0bd-42de-8108-45c128b023ed nodeName:}" failed. No retries permitted until 2026-07-31 14:12:24.603143363 +0000 UTC m=+10.111522425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2664k" (UniqueName: "kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k") pod "network-check-target-x2qjh" (UID: "a72ae63b-a0bd-42de-8108-45c128b023ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:20.977263 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:20.977184 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:20.977263 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:20.977227 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:20.977490 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.977343 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:20.977542 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:20.977496 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:21.977651 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:21.977587 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:21.978137 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:21.977724 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:22.977540 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:22.977474 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:22.977723 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:22.977618 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:22.977723 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:22.977657 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:22.978154 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:22.977745 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:23.977585 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:23.977485 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:23.977739 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:23.977634 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:24.535684 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:24.535631 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:24.535869 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:24.535729 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:24.535869 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.535804 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:24.535869 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.535830 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:24.536055 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.535891 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret podName:ce208340-fc19-4886-b1ce-1e4d7807dbe2 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:32.535870879 +0000 UTC m=+18.044249929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret") pod "global-pull-secret-syncer-vlwj7" (UID: "ce208340-fc19-4886-b1ce-1e4d7807dbe2") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:24.536055 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.535930 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:32.535917032 +0000 UTC m=+18.044296082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:24.636659 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:24.636609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:24.636864 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.636745 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:24.636864 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.636764 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:24.636864 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.636775 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2664k for pod openshift-network-diagnostics/network-check-target-x2qjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:24.636864 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.636833 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k podName:a72ae63b-a0bd-42de-8108-45c128b023ed nodeName:}" failed. No retries permitted until 2026-07-31 14:12:32.636813973 +0000 UTC m=+18.145193024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2664k" (UniqueName: "kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k") pod "network-check-target-x2qjh" (UID: "a72ae63b-a0bd-42de-8108-45c128b023ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:24.978809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:24.978556 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:24.978809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:24.978659 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:24.978809 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.978689 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:24.978809 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:24.978732 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:25.977852 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:25.977771 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:25.978022 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:25.977916 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:26.977706 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:26.977677 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:26.978199 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:26.977821 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:26.978335 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:26.978314 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:26.978452 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:26.978429 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:27.977442 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:27.977410 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:27.977623 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:27.977541 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:28.977397 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:28.977366 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:28.977397 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:28.977403 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:28.977920 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:28.977494 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:28.977920 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:28.977622 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:29.977561 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:29.977521 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:29.978001 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:29.977640 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:30.977411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:30.977370 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:30.977569 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:30.977528 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:30.977641 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:30.977594 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:30.977938 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:30.977695 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:31.977296 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:31.977260 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:31.977514 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:31.977386 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:32.597154 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:32.597112 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:32.597616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:32.597206 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:32.597616 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.597295 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:32.597616 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.597332 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:32.597616 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.597372 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret podName:ce208340-fc19-4886-b1ce-1e4d7807dbe2 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:48.597355857 +0000 UTC m=+34.105734905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret") pod "global-pull-secret-syncer-vlwj7" (UID: "ce208340-fc19-4886-b1ce-1e4d7807dbe2") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:32.597616 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.597396 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:48.597379024 +0000 UTC m=+34.105758071 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:32.698641 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:32.698562 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:32.698827 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.698696 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:32.698827 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.698718 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:32.698827 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.698730 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2664k for pod openshift-network-diagnostics/network-check-target-x2qjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:32.698827 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.698793 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k podName:a72ae63b-a0bd-42de-8108-45c128b023ed nodeName:}" failed. No retries permitted until 2026-07-31 14:12:48.698775815 +0000 UTC m=+34.207154874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2664k" (UniqueName: "kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k") pod "network-check-target-x2qjh" (UID: "a72ae63b-a0bd-42de-8108-45c128b023ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:32.977821 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:32.977735 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:32.977821 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:32.977774 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:32.978049 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.977858 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:32.978049 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:32.978027 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:33.977764 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:33.977722 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:33.978228 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:33.977856 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:34.978206 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:34.978172 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:34.978678 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:34.978297 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:34.978678 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:34.978348 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:34.978678 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:34.978456 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:35.977248 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:35.977216 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:35.977426 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:35.977343 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:36.978053 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:36.977546 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:36.978664 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:36.977606 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:36.978664 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:36.978140 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:36.978664 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:36.978175 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:37.057798 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.057767 2558 generic.go:356] "Generic (PLEG): container finished" podID="70381455-7d7d-4682-9ad3-0c19ed448194" containerID="6580a6463bb1926c505be6609d3aefe3753933d0459fe917571f8eac9c081d79" exitCode=0 Jul 31 14:12:37.057973 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.057839 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" event={"ID":"70381455-7d7d-4682-9ad3-0c19ed448194","Type":"ContainerDied","Data":"6580a6463bb1926c505be6609d3aefe3753933d0459fe917571f8eac9c081d79"} Jul 31 14:12:37.059495 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.059472 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" event={"ID":"2f51637b-25e7-4268-a752-37b03d2a44d9","Type":"ContainerStarted","Data":"9a808b91a40c861e27db943333b0efd8cc88d4d69069c26cbb55be9ec4b7ca5a"} Jul 31 14:12:37.060887 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.060855 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5t4bh" event={"ID":"9b6b34e3-8489-4f7e-aee5-df61d5cf87cc","Type":"ContainerStarted","Data":"a343ad90ab882415a96b6d5b7d093d3d8c6be7a567b94d273a701e42c82bb871"} Jul 31 14:12:37.064133 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.064101 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"69fc1296ae0c7f370c95b42c88ad3e3daf1e5d61d9af4f297b95ad1818ef5f2e"} Jul 31 14:12:37.064230 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.064177 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"3cac5223f886e5e9b86d6adb4fde73e6f6eaf0dbb434589e16b53d38c13b4258"} Jul 31 14:12:37.064290 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.064230 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"e498cfe1d753b43b63deafb5046be444fcea4e1d741b457a8b5781d39699e3bd"} Jul 31 14:12:37.064290 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.064277 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"ccd09597ee4a0960414412d9277a686d70570a6e101ff40ffa499b9523b87595"} Jul 31 14:12:37.064389 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.064313 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"68c71c7d03e8849261e28e60ef8d3758086d63c8419e2ba5b6cc53c0510cd960"} Jul 31 14:12:37.064389 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.064352 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"643c3459a8c9656d0c0bf02ea8fe22fed9d1773b0749fd8470b35a5ee7ffa76b"} Jul 31 14:12:37.065518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.065496 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kd9wm" event={"ID":"4c0cf086-f032-4b04-adcb-23df0705d67c","Type":"ContainerStarted","Data":"b3d824002e9d129bbe18e3380a91ad468cc7a804875629d6a43644f3dff3439e"} Jul 31 14:12:37.066780 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.066760 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4gd4d" event={"ID":"d57b476a-2d1a-4caf-be2c-ea4677ca3602","Type":"ContainerStarted","Data":"3bf68698115faf0aecd4f279d73042c8de80878f7cd95d1b4710405579958795"} Jul 31 14:12:37.068050 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.068030 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" event={"ID":"c9d45a40-0428-4335-b168-eaf9a86789ae","Type":"ContainerStarted","Data":"0e4661a93c09a30f0be6322a2c461c2af3a7e2d7aee83596f0740bedf215866f"} Jul 31 14:12:37.069318 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.069300 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8wjhs" event={"ID":"809b2748-9aad-4e40-864e-2252739a4f03","Type":"ContainerStarted","Data":"4b531e5e9ffd6fdb039ad25d236f0daafee8dc04fd5c51278afc630bcf0e8294"} Jul 31 14:12:37.079225 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.079178 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-33.ec2.internal" podStartSLOduration=22.079167162 podStartE2EDuration="22.079167162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:20.038611151 +0000 UTC m=+5.546990219" watchObservedRunningTime="2026-07-31 14:12:37.079167162 +0000 UTC m=+22.587546227" Jul 31 14:12:37.091337 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.091300 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5t4bh" podStartSLOduration=11.71822066 podStartE2EDuration="22.09128731s" totalImagesPullingTime="10.37306665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:37.090991758 +0000 UTC m=+22.599370823" watchObservedRunningTime="2026-07-31 14:12:37.09128731 +0000 UTC m=+22.599666376" Jul 31 14:12:37.106624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.106587 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f8tr8" podStartSLOduration=3.839625541 podStartE2EDuration="22.106576581s" totalImagesPullingTime="18.26695104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:37.106398184 +0000 UTC m=+22.614777250" watchObservedRunningTime="2026-07-31 14:12:37.106576581 +0000 UTC m=+22.614955646" Jul 31 14:12:37.120678 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.120640 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8wjhs" podStartSLOduration=3.853508954 podStartE2EDuration="22.120628348s" totalImagesPullingTime="18.267119394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:37.120493494 +0000 UTC m=+22.628872560" watchObservedRunningTime="2026-07-31 14:12:37.120628348 +0000 UTC m=+22.629007414" Jul 31 14:12:37.163262 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.163225 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4gd4d" podStartSLOduration=3.831269398 podStartE2EDuration="22.163213523s" totalImagesPullingTime="18.331944125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:37.148035915 +0000 UTC m=+22.656414981" watchObservedRunningTime="2026-07-31 14:12:37.163213523 +0000 UTC m=+22.671592582" Jul 31 14:12:37.163548 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.163523 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kd9wm" podStartSLOduration=3.900859836 podStartE2EDuration="22.163516202s" totalImagesPullingTime="18.262656366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:37.163093706 +0000 UTC m=+22.671472773" watchObservedRunningTime="2026-07-31 14:12:37.163516202 +0000 UTC m=+22.671895267" Jul 31 14:12:37.977606 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:37.977583 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:37.977733 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:37.977714 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:38.015404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.015376 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 31 14:12:38.073647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.073600 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" event={"ID":"2f51637b-25e7-4268-a752-37b03d2a44d9","Type":"ContainerStarted","Data":"75ee9bddf47b52d39b41260499dcb6b4dc98b744c4d86ad16639d6aaecc9777c"} Jul 31 14:12:38.075100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.075072 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dzxbx" event={"ID":"c5ea2efc-9e78-45d5-a382-7233eab15fce","Type":"ContainerStarted","Data":"85a27d30fef21f1865067c37f31f9a7e8a80bfb006745c1547aefd533deac400"} Jul 31 14:12:38.089426 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.089386 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dzxbx" podStartSLOduration=4.82018196 podStartE2EDuration="23.089372113s" totalImagesPullingTime="18.269190153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:38.088878049 +0000 UTC m=+23.597257118" watchObservedRunningTime="2026-07-31 14:12:38.089372113 +0000 UTC m=+23.597751179" Jul 31 14:12:38.911284 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.911181 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:12:38.01539906Z","UUID":"9c681eea-0bc3-47ef-85b8-442813e6595d","Handler":null,"Name":"","Endpoint":""} Jul 31 14:12:38.913812 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.913792 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 31 14:12:38.913952 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.913842 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 31 14:12:38.981650 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.981621 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:38.981827 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:38.981764 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:38.981827 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:38.981797 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:38.981971 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:38.981864 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:39.084060 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:39.083975 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"8ae61ae0c249f301911d2e989cba51617872a4255e6bdc20432c5dafc2f4f2b6"} Jul 31 14:12:39.141374 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:39.141340 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:39.764178 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:39.764013 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:39.764662 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:39.764645 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:39.977284 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:39.977138 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:39.977284 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:39.977256 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:40.088508 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:40.088468 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" event={"ID":"2f51637b-25e7-4268-a752-37b03d2a44d9","Type":"ContainerStarted","Data":"42b609d7871bb9e1c78210572243e98ee5474d5dbcf5b33a631e4a61d68518e8"} Jul 31 14:12:40.089281 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:40.089258 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5t4bh" Jul 31 14:12:40.109462 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:40.109412 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp9cn" podStartSLOduration=3.71919815 podStartE2EDuration="25.109393229s" totalImagesPullingTime="21.390195079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:40.10876946 +0000 UTC m=+25.617148525" watchObservedRunningTime="2026-07-31 14:12:40.109393229 +0000 UTC m=+25.617772291" Jul 31 14:12:40.977565 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:40.977537 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:40.977706 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:40.977539 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:40.977706 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:40.977659 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:40.977826 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:40.977743 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:41.096334 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:41.095662 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" event={"ID":"b8dd22be-cc3b-43b6-9dcd-61faa79da219","Type":"ContainerStarted","Data":"7df1e89dad27db60ca475c5a547e6adecc788fbd07d72bf78d46d22b7d428475"} Jul 31 14:12:41.123688 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:41.123638 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" podStartSLOduration=7.795410319 podStartE2EDuration="26.123617024s" totalImagesPullingTime="18.328206705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:41.123039306 +0000 UTC m=+26.631418391" watchObservedRunningTime="2026-07-31 14:12:41.123617024 +0000 UTC m=+26.631996105" Jul 31 14:12:41.977887 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:41.977855 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:41.978042 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:41.977992 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:42.098287 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:42.098256 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:42.098869 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:42.098369 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:42.098869 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:42.098421 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:42.115572 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:42.115549 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:42.115705 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:42.115663 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:12:42.979949 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:42.979757 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:42.980109 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:42.979813 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:42.980109 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:42.980034 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:42.980185 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:42.980106 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:43.037586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:43.037560 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kdtjx"] Jul 31 14:12:43.040264 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:43.040240 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x2qjh"] Jul 31 14:12:43.040365 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:43.040339 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:43.040452 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:43.040433 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:43.051811 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:43.051787 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vlwj7"] Jul 31 14:12:43.099269 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:43.099246 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:43.099565 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:43.099246 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:43.099565 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:43.099389 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:43.099565 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:43.099437 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:44.103440 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:44.103361 2558 generic.go:356] "Generic (PLEG): container finished" podID="70381455-7d7d-4682-9ad3-0c19ed448194" containerID="8f03c11356e1536d03086440a956ade817f73a236b91df3381a1e615c927563d" exitCode=0 Jul 31 14:12:44.103790 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:44.103440 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" event={"ID":"70381455-7d7d-4682-9ad3-0c19ed448194","Type":"ContainerDied","Data":"8f03c11356e1536d03086440a956ade817f73a236b91df3381a1e615c927563d"} Jul 31 14:12:44.981214 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:44.981178 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:44.981404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:44.981293 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:44.981404 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:44.981302 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:44.981404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:44.981324 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:44.981512 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:44.981420 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:44.981512 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:44.981491 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:46.110525 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:46.110485 2558 generic.go:356] "Generic (PLEG): container finished" podID="70381455-7d7d-4682-9ad3-0c19ed448194" containerID="bff37dc6ce84a07c698d5787d60a0a693497073927c7e6943d456cdf97463c0b" exitCode=0 Jul 31 14:12:46.110891 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:46.110534 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" event={"ID":"70381455-7d7d-4682-9ad3-0c19ed448194","Type":"ContainerDied","Data":"bff37dc6ce84a07c698d5787d60a0a693497073927c7e6943d456cdf97463c0b"} Jul 31 14:12:46.980247 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:46.980223 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:46.980410 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:46.980300 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:46.980410 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:46.980326 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:46.980487 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:46.980455 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:46.980538 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:46.980494 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:46.980586 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:46.980552 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:48.120004 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:48.119964 2558 generic.go:356] "Generic (PLEG): container finished" podID="70381455-7d7d-4682-9ad3-0c19ed448194" containerID="72b74355148b3f952e36b955ed6dcbe3d2def99559df476c9dab1187892465cc" exitCode=0 Jul 31 14:12:48.120372 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:48.120035 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" event={"ID":"70381455-7d7d-4682-9ad3-0c19ed448194","Type":"ContainerDied","Data":"72b74355148b3f952e36b955ed6dcbe3d2def99559df476c9dab1187892465cc"} Jul 31 14:12:48.615621 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:48.615593 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:48.615762 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:48.615648 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:48.615762 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.615736 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:48.615880 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.615797 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:20.615783431 +0000 UTC m=+66.124162479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 31 14:12:48.615880 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.615736 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:48.615880 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.615872 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret podName:ce208340-fc19-4886-b1ce-1e4d7807dbe2 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:20.615858491 +0000 UTC m=+66.124237535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret") pod "global-pull-secret-syncer-vlwj7" (UID: "ce208340-fc19-4886-b1ce-1e4d7807dbe2") : object "kube-system"/"original-pull-secret" not registered Jul 31 14:12:48.716533 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:48.716498 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:48.716701 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.716643 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 31 14:12:48.716701 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.716670 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 31 14:12:48.716701 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.716684 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2664k for pod openshift-network-diagnostics/network-check-target-x2qjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:48.716815 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.716748 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k podName:a72ae63b-a0bd-42de-8108-45c128b023ed nodeName:}" failed. No retries permitted until 2026-07-31 14:13:20.716729319 +0000 UTC m=+66.225108368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2664k" (UniqueName: "kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k") pod "network-check-target-x2qjh" (UID: "a72ae63b-a0bd-42de-8108-45c128b023ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 31 14:12:48.977288 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:48.977211 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:48.977288 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:48.977226 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:48.977475 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:48.977209 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:48.977475 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.977316 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2qjh" podUID="a72ae63b-a0bd-42de-8108-45c128b023ed" Jul 31 14:12:48.977475 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.977380 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vlwj7" podUID="ce208340-fc19-4886-b1ce-1e4d7807dbe2" Jul 31 14:12:48.977592 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:48.977478 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:12:49.793597 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.793572 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-33.ec2.internal" event="NodeReady" Jul 31 14:12:49.794208 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.793715 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 31 14:12:49.826801 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.826774 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh"] Jul 31 14:12:49.828718 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.828697 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d79fc885d-lrzsb"] Jul 31 14:12:49.828855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.828837 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:49.830827 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.830807 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.831368 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.831347 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 31 14:12:49.831809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.831757 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 31 14:12:49.831809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.831760 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jrsss\"" Jul 31 14:12:49.834068 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.833947 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9w89q\"" Jul 31 14:12:49.834161 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.834089 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 31 14:12:49.834161 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.834100 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 31 14:12:49.834286 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.834267 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 31 14:12:49.845192 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.845075 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh"] Jul 31 14:12:49.846790 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.845793 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 31 14:12:49.846790 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.846260 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nwlqc"] Jul 31 14:12:49.849384 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.849361 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d79fc885d-lrzsb"] Jul 31 14:12:49.849500 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.849396 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bzp46"] Jul 31 14:12:49.849562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.849498 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:49.852097 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.852078 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9vdf\"" Jul 31 14:12:49.852307 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.852288 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 31 14:12:49.852423 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.852361 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 31 14:12:49.854653 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.854634 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bzp46" Jul 31 14:12:49.856836 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.856796 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nwlqc"] Jul 31 14:12:49.857306 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.857238 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 31 14:12:49.858220 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.858055 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p49zl\"" Jul 31 14:12:49.858220 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.858068 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 31 14:12:49.858372 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.858249 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 31 14:12:49.868867 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.868848 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bzp46"] Jul 31 14:12:49.924801 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.924770 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnvzp\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-kube-api-access-tnvzp\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.925004 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.924821 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.925004 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.924852 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b85ca420-fd83-4ed0-90b6-609395bc209c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:49.925004 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.924876 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-image-registry-private-configuration\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.925004 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.924896 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-trusted-ca\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.925004 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.924967 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e6c2de2-4244-4337-88b5-80176d3b4681-ca-trust-extracted\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.925256 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.925039 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-certificates\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.925256 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.925086 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:49.925256 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.925129 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-installation-pull-secrets\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.925256 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.925161 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-bound-sa-token\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:49.974364 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:49.974307 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:50.026192 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026161 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-certificates\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.026376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026200 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/697d6e95-2340-45cf-8c34-626bd16dbd73-config-volume\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.026376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026235 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:50.026376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026253 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxc9x\" (UniqueName: \"kubernetes.io/projected/eabbfc47-226f-4ed0-a0aa-a495528d9f29-kube-api-access-fxc9x\") pod \"ingress-canary-bzp46\" (UID: \"eabbfc47-226f-4ed0-a0aa-a495528d9f29\") " pod="openshift-ingress-canary/ingress-canary-bzp46" Jul 31 14:12:50.026376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026311 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-installation-pull-secrets\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.026376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026335 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-bound-sa-token\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.026376 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.026338 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:12:50.026376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026362 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnvzp\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-kube-api-access-tnvzp\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.026411 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:12:50.526388544 +0000 UTC m=+36.034767588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026452 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026481 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026507 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697d6e95-2340-45cf-8c34-626bd16dbd73-tmp-dir\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026544 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp92n\" (UniqueName: \"kubernetes.io/projected/697d6e95-2340-45cf-8c34-626bd16dbd73-kube-api-access-dp92n\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.026572 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.026586 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026586 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b85ca420-fd83-4ed0-90b6-609395bc209c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026616 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-image-registry-private-configuration\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.026643 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:50.526624862 +0000 UTC m=+36.035003921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026682 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eabbfc47-226f-4ed0-a0aa-a495528d9f29-cert\") pod \"ingress-canary-bzp46\" (UID: \"eabbfc47-226f-4ed0-a0aa-a495528d9f29\") " pod="openshift-ingress-canary/ingress-canary-bzp46" Jul 31 14:12:50.026731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026721 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-trusted-ca\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.027335 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026754 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e6c2de2-4244-4337-88b5-80176d3b4681-ca-trust-extracted\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.027335 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.026818 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-certificates\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.027335 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.027143 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e6c2de2-4244-4337-88b5-80176d3b4681-ca-trust-extracted\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.027335 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.027244 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b85ca420-fd83-4ed0-90b6-609395bc209c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:50.027657 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.027632 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-trusted-ca\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.030545 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.030520 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-image-registry-private-configuration\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.030545 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.030530 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-installation-pull-secrets\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.034330 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.034299 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnvzp\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-kube-api-access-tnvzp\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.037332 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.037309 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-bound-sa-token\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.060294 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.060265 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:50.128025 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.127994 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxc9x\" (UniqueName: \"kubernetes.io/projected/eabbfc47-226f-4ed0-a0aa-a495528d9f29-kube-api-access-fxc9x\") pod \"ingress-canary-bzp46\" (UID: \"eabbfc47-226f-4ed0-a0aa-a495528d9f29\") " pod="openshift-ingress-canary/ingress-canary-bzp46" Jul 31 14:12:50.128208 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.128173 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.128275 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.128218 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697d6e95-2340-45cf-8c34-626bd16dbd73-tmp-dir\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.128332 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.128309 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:12:50.128383 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.128343 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp92n\" (UniqueName: \"kubernetes.io/projected/697d6e95-2340-45cf-8c34-626bd16dbd73-kube-api-access-dp92n\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.128383 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.128378 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:50.628356918 +0000 UTC m=+36.136735966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:12:50.128504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.128421 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eabbfc47-226f-4ed0-a0aa-a495528d9f29-cert\") pod \"ingress-canary-bzp46\" (UID: \"eabbfc47-226f-4ed0-a0aa-a495528d9f29\") " pod="openshift-ingress-canary/ingress-canary-bzp46" Jul 31 14:12:50.128504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.128470 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/697d6e95-2340-45cf-8c34-626bd16dbd73-config-volume\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.128675 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.128657 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697d6e95-2340-45cf-8c34-626bd16dbd73-tmp-dir\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.129018 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.128997 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/697d6e95-2340-45cf-8c34-626bd16dbd73-config-volume\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.131344 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.131320 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eabbfc47-226f-4ed0-a0aa-a495528d9f29-cert\") pod \"ingress-canary-bzp46\" (UID: \"eabbfc47-226f-4ed0-a0aa-a495528d9f29\") " pod="openshift-ingress-canary/ingress-canary-bzp46" Jul 31 14:12:50.138005 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.137982 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxc9x\" (UniqueName: \"kubernetes.io/projected/eabbfc47-226f-4ed0-a0aa-a495528d9f29-kube-api-access-fxc9x\") pod \"ingress-canary-bzp46\" (UID: \"eabbfc47-226f-4ed0-a0aa-a495528d9f29\") " pod="openshift-ingress-canary/ingress-canary-bzp46" Jul 31 14:12:50.138180 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.138160 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp92n\" (UniqueName: \"kubernetes.io/projected/697d6e95-2340-45cf-8c34-626bd16dbd73-kube-api-access-dp92n\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.168480 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.168458 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bzp46" Jul 31 14:12:50.323857 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.323816 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bzp46"] Jul 31 14:12:50.327781 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:12:50.327753 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabbfc47_226f_4ed0_a0aa_a495528d9f29.slice/crio-fb8dca99cd22ad53307acfbc47ad545c5700a74df47155fad2e75661101904b4 WatchSource:0}: Error finding container fb8dca99cd22ad53307acfbc47ad545c5700a74df47155fad2e75661101904b4: Status 404 returned error can't find the container with id fb8dca99cd22ad53307acfbc47ad545c5700a74df47155fad2e75661101904b4 Jul 31 14:12:50.532553 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.532329 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:50.532711 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.532565 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:50.532711 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.532475 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:12:50.532711 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.532593 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:12:50.532711 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.532654 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:51.532635524 +0000 UTC m=+37.041014581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:12:50.532711 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.532687 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:12:50.532993 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.532739 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:12:51.532724399 +0000 UTC m=+37.041103443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:12:50.633795 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.633766 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:50.633983 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.633928 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:12:50.634023 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:50.633991 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:51.633971551 +0000 UTC m=+37.142350595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:12:50.867607 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.867569 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:50.922591 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.922536 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:50.977255 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.977224 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:12:50.977255 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.977248 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:12:50.977449 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.977224 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:12:50.981534 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.981510 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 14:12:50.981666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.981536 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 14:12:50.981666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.981544 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lwbh\"" Jul 31 14:12:50.981666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.981556 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 14:12:50.981666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.981511 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 14:12:50.996548 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:50.996362 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rdxfb\"" Jul 31 14:12:51.128951 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:51.128856 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bzp46" event={"ID":"eabbfc47-226f-4ed0-a0aa-a495528d9f29","Type":"ContainerStarted","Data":"fb8dca99cd22ad53307acfbc47ad545c5700a74df47155fad2e75661101904b4"} Jul 31 14:12:51.540796 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:51.540721 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:51.540971 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:51.540822 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:51.541040 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:51.540994 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:12:51.541040 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:51.541017 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:12:51.541040 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:51.541026 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:12:51.541195 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:51.541122 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:53.541067458 +0000 UTC m=+39.049446503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:12:51.541195 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:51.541175 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:12:53.541162973 +0000 UTC m=+39.049542019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:12:51.642061 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:51.642026 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:51.642318 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:51.642216 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:12:51.642318 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:51.642300 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:53.642278562 +0000 UTC m=+39.150657624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:12:51.803956 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:51.803877 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:51.904696 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:51.904659 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:53.560727 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:53.560687 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:53.561243 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:53.560794 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:53.561243 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:53.560876 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:12:53.561243 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:53.560915 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:12:53.561243 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:53.560932 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:12:53.561243 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:53.560989 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:57.560966104 +0000 UTC m=+43.069345169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:12:53.561243 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:53.561009 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:12:57.560999372 +0000 UTC m=+43.069378421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:12:53.661804 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:53.661763 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:53.662007 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:53.661953 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:12:53.662059 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:53.662021 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:57.662004047 +0000 UTC m=+43.170383091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:12:54.239334 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:54.239286 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:54.572400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:54.572326 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:55.144299 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:55.144263 2558 generic.go:356] "Generic (PLEG): container finished" podID="70381455-7d7d-4682-9ad3-0c19ed448194" containerID="bfeeddf78c67c8ac6a041f06a607dbc641ef280f3e141e66dff48e27e50d3fde" exitCode=0 Jul 31 14:12:55.144496 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:55.144348 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" event={"ID":"70381455-7d7d-4682-9ad3-0c19ed448194","Type":"ContainerDied","Data":"bfeeddf78c67c8ac6a041f06a607dbc641ef280f3e141e66dff48e27e50d3fde"} Jul 31 14:12:55.145807 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:55.145779 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bzp46" event={"ID":"eabbfc47-226f-4ed0-a0aa-a495528d9f29","Type":"ContainerStarted","Data":"76d86bf05bcbea0d0e0a47f9cfc7dc2938fb04d071a1db72c7811e1fd33075c4"} Jul 31 14:12:56.150859 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:56.150823 2558 generic.go:356] "Generic (PLEG): container finished" podID="70381455-7d7d-4682-9ad3-0c19ed448194" containerID="e64efdc610373a4887d8be4cd1b7a3ff437a51868b861d05de4332ff40538648" exitCode=0 Jul 31 14:12:56.151235 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:56.150928 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" event={"ID":"70381455-7d7d-4682-9ad3-0c19ed448194","Type":"ContainerDied","Data":"e64efdc610373a4887d8be4cd1b7a3ff437a51868b861d05de4332ff40538648"} Jul 31 14:12:56.174080 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:56.174000 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bzp46" podStartSLOduration=3.25312261 podStartE2EDuration="7.173986712s" totalImagesPullingTime="3.920864102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:55.183976781 +0000 UTC m=+40.692355858" watchObservedRunningTime="2026-07-31 14:12:56.173986712 +0000 UTC m=+41.682365777" Jul 31 14:12:56.870043 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:56.869990 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:56.920999 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:56.920963 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:57.156580 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:57.156505 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" event={"ID":"70381455-7d7d-4682-9ad3-0c19ed448194","Type":"ContainerStarted","Data":"ce10c7f6a31dbf2e3b080b7154780caa7a96a0e6592a615b7fa7ca5c627ba7cc"} Jul 31 14:12:57.178981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:57.178924 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b8tpw" podStartSLOduration=17.790575049 podStartE2EDuration="42.178883963s" totalImagesPullingTime="24.388308914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:57.177366386 +0000 UTC m=+42.685745453" watchObservedRunningTime="2026-07-31 14:12:57.178883963 +0000 UTC m=+42.687263029" Jul 31 14:12:57.300683 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:57.300647 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:57.356362 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:57.356326 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:12:57.587625 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:57.587532 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:12:57.587625 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:57.587602 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:12:57.587847 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:57.587684 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:12:57.587847 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:57.587694 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:12:57.587847 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:57.587716 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:12:57.587847 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:57.587748 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:13:05.58773353 +0000 UTC m=+51.096112575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:12:57.587847 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:57.587762 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:05.587755513 +0000 UTC m=+51.096134556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:12:57.688144 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:12:57.688107 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:12:57.688315 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:57.688240 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:12:57.688315 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:12:57.688296 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:05.688281127 +0000 UTC m=+51.196660171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:13:02.917822 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:02.917788 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:03.828343 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:03.828302 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:03.928921 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:03.928868 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:03.979585 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:03.979547 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:04.948244 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:04.948206 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:05.039715 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:05.039681 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:05.643196 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:05.643155 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:13:05.643514 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:05.643231 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:13:05.643514 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:05.643323 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:13:05.643514 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:05.643338 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:13:05.643514 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:05.643363 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:13:05.643514 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:05.643390 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:13:21.643374461 +0000 UTC m=+67.151753505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:13:05.643514 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:05.643434 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:21.64341611 +0000 UTC m=+67.151795168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:13:05.744242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:05.744206 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:13:05.744384 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:05.744319 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:05.744384 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:05.744367 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:21.74435471 +0000 UTC m=+67.252733754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:13:06.774071 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:06.774032 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:07.881045 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:07.881010 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:07.982022 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:07.981979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:08.083137 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:08.083107 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:10.095326 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.095297 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv"] Jul 31 14:13:10.097356 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.097338 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.099917 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.099880 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 31 14:13:10.103682 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.101528 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 31 14:13:10.103682 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.101588 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 31 14:13:10.103682 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.103412 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 31 14:13:10.108236 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.108209 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv"] Jul 31 14:13:10.176285 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.176256 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7eae1fb9-92c1-4ab0-a455-73811336e4b2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.176411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.176290 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eae1fb9-92c1-4ab0-a455-73811336e4b2-tmp\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.176411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.176309 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2wn\" (UniqueName: \"kubernetes.io/projected/7eae1fb9-92c1-4ab0-a455-73811336e4b2-kube-api-access-9t2wn\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.276897 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.276866 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eae1fb9-92c1-4ab0-a455-73811336e4b2-tmp\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.277026 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.276951 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2wn\" (UniqueName: \"kubernetes.io/projected/7eae1fb9-92c1-4ab0-a455-73811336e4b2-kube-api-access-9t2wn\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.277081 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.277069 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7eae1fb9-92c1-4ab0-a455-73811336e4b2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.277144 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.277127 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eae1fb9-92c1-4ab0-a455-73811336e4b2-tmp\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.280266 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.280238 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7eae1fb9-92c1-4ab0-a455-73811336e4b2-klusterlet-config\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.285084 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.285063 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2wn\" (UniqueName: \"kubernetes.io/projected/7eae1fb9-92c1-4ab0-a455-73811336e4b2-kube-api-access-9t2wn\") pod \"klusterlet-addon-workmgr-6968d6c77-cb4tv\" (UID: \"7eae1fb9-92c1-4ab0-a455-73811336e4b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.408397 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.408360 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:10.523968 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:10.523943 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv"] Jul 31 14:13:10.526747 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:13:10.526714 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eae1fb9_92c1_4ab0_a455_73811336e4b2.slice/crio-44397a9fb51740145c2dde9bc0d182af2f84fbc0e5bf505654eb302a8ddf2f2d WatchSource:0}: Error finding container 44397a9fb51740145c2dde9bc0d182af2f84fbc0e5bf505654eb302a8ddf2f2d: Status 404 returned error can't find the container with id 44397a9fb51740145c2dde9bc0d182af2f84fbc0e5bf505654eb302a8ddf2f2d Jul 31 14:13:11.188216 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:11.188179 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" event={"ID":"7eae1fb9-92c1-4ab0-a455-73811336e4b2","Type":"ContainerStarted","Data":"44397a9fb51740145c2dde9bc0d182af2f84fbc0e5bf505654eb302a8ddf2f2d"} Jul 31 14:13:14.117503 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:14.117474 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9hbr" Jul 31 14:13:14.272290 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:14.272258 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:14.323993 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:14.323952 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:15.200884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:15.200848 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" event={"ID":"7eae1fb9-92c1-4ab0-a455-73811336e4b2","Type":"ContainerStarted","Data":"844c9169ff8358316f7dd314ca86678ec8eb0036e1e8e80eef8b85dd5e550d10"} Jul 31 14:13:15.201299 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:15.201053 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:15.202757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:15.202736 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:13:15.216283 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:15.216240 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" podStartSLOduration=1.110114383 podStartE2EDuration="5.216227673s" totalImagesPullingTime="4.10611329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:15.215658717 +0000 UTC m=+60.724037784" watchObservedRunningTime="2026-07-31 14:13:15.216227673 +0000 UTC m=+60.724606738" Jul 31 14:13:15.670612 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:15.670577 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:19.900407 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:19.900368 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:20.002285 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.002246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:20.655215 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.655183 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:13:20.655381 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.655256 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:13:20.658433 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.658413 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 31 14:13:20.658479 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.658413 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 31 14:13:20.665876 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:20.665855 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 14:13:20.665939 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:20.665929 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:14:24.665899827 +0000 UTC m=+130.174278871 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : secret "metrics-daemon-secret" not found Jul 31 14:13:20.668018 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.667987 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce208340-fc19-4886-b1ce-1e4d7807dbe2-original-pull-secret\") pod \"global-pull-secret-syncer-vlwj7\" (UID: \"ce208340-fc19-4886-b1ce-1e4d7807dbe2\") " pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:13:20.705258 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.705236 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vlwj7" Jul 31 14:13:20.756600 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.756567 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:13:20.760124 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.760099 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 31 14:13:20.770296 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.770268 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 31 14:13:20.781000 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.780933 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2664k\" (UniqueName: \"kubernetes.io/projected/a72ae63b-a0bd-42de-8108-45c128b023ed-kube-api-access-2664k\") pod \"network-check-target-x2qjh\" (UID: \"a72ae63b-a0bd-42de-8108-45c128b023ed\") " pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:13:20.825858 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.825831 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vlwj7"] Jul 31 14:13:20.828560 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:13:20.828531 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce208340_fc19_4886_b1ce_1e4d7807dbe2.slice/crio-fff7d93ac4213ef50e40544b229b84851442fd1f85318d9ac1faffbe26c14db3 WatchSource:0}: Error finding container fff7d93ac4213ef50e40544b229b84851442fd1f85318d9ac1faffbe26c14db3: Status 404 returned error can't find the container with id fff7d93ac4213ef50e40544b229b84851442fd1f85318d9ac1faffbe26c14db3 Jul 31 14:13:20.926020 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.925952 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:20.981207 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.981172 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:20.993792 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:20.993770 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rdxfb\"" Jul 31 14:13:21.001250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:21.001228 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:13:21.113943 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:21.113897 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x2qjh"] Jul 31 14:13:21.116843 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:13:21.116812 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72ae63b_a0bd_42de_8108_45c128b023ed.slice/crio-ae6e0112d002d13c9b9b942c2047069e766cdeb9f78bedea17508e96bf9d77e1 WatchSource:0}: Error finding container ae6e0112d002d13c9b9b942c2047069e766cdeb9f78bedea17508e96bf9d77e1: Status 404 returned error can't find the container with id ae6e0112d002d13c9b9b942c2047069e766cdeb9f78bedea17508e96bf9d77e1 Jul 31 14:13:21.215731 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:21.215661 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x2qjh" event={"ID":"a72ae63b-a0bd-42de-8108-45c128b023ed","Type":"ContainerStarted","Data":"ae6e0112d002d13c9b9b942c2047069e766cdeb9f78bedea17508e96bf9d77e1"} Jul 31 14:13:21.216666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:21.216638 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vlwj7" event={"ID":"ce208340-fc19-4886-b1ce-1e4d7807dbe2","Type":"ContainerStarted","Data":"fff7d93ac4213ef50e40544b229b84851442fd1f85318d9ac1faffbe26c14db3"} Jul 31 14:13:21.663110 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:21.663074 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:13:21.663294 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:21.663145 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:13:21.663294 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:21.663232 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:13:21.663378 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:21.663302 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:13:53.663287318 +0000 UTC m=+99.171666366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:13:21.663378 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:21.663235 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:13:21.663378 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:21.663341 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:13:21.663477 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:21.663391 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:53.663378291 +0000 UTC m=+99.171757353 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:13:21.763541 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:21.763504 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:13:21.763695 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:21.763620 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:21.763695 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:21.763682 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:53.763663185 +0000 UTC m=+99.272042231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:13:27.234313 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:27.234277 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vlwj7" event={"ID":"ce208340-fc19-4886-b1ce-1e4d7807dbe2","Type":"ContainerStarted","Data":"8e7f91c11fab01997043ddf35138c387f6df0a5ec7f7f1057aba105c4bf7e299"} Jul 31 14:13:27.235750 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:27.235729 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x2qjh" event={"ID":"a72ae63b-a0bd-42de-8108-45c128b023ed","Type":"ContainerStarted","Data":"c4ec89f371e12603188f55f9e29ae348e723bb40b9ac76ed54868d78ed717822"} Jul 31 14:13:27.235873 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:27.235859 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:13:27.249674 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:27.249631 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vlwj7" podStartSLOduration=66.837962731 podStartE2EDuration="1m12.24961734s" totalImagesPullingTime="5.411654609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:27.249269387 +0000 UTC m=+72.757648453" watchObservedRunningTime="2026-07-31 14:13:27.24961734 +0000 UTC m=+72.757996405" Jul 31 14:13:27.269142 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:27.269107 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x2qjh" podStartSLOduration=67.157406561 podStartE2EDuration="1m12.269095487s" totalImagesPullingTime="5.111688926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:27.268541881 +0000 UTC m=+72.776920948" watchObservedRunningTime="2026-07-31 14:13:27.269095487 +0000 UTC m=+72.777474590" Jul 31 14:13:29.867089 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:29.867055 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:29.922899 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:29.922864 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:41.867779 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:41.867643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:41.916683 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:41.916649 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:41.967368 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:41.967336 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:47.191559 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:47.191511 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:13:53.711546 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:53.711503 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:13:53.712050 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:53.711567 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:13:53.712050 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:53.711673 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:13:53.712050 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:53.711748 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:14:57.711733986 +0000 UTC m=+163.220113033 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:13:53.712050 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:53.711676 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:13:53.712050 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:53.711783 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:13:53.712050 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:53.711827 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:14:57.711812847 +0000 UTC m=+163.220191909 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:13:53.812884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:53.812856 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:13:53.813031 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:53.813012 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:13:53.813078 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:13:53.813072 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:14:57.813057584 +0000 UTC m=+163.321436628 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:13:58.241184 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:13:58.241147 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x2qjh" Jul 31 14:14:17.863052 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:17.863018 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:14:18.540887 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:18.540850 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:14:24.722716 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:24.722663 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:14:24.723253 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:24.722844 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 14:14:24.723253 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:24.722951 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:26.72292781 +0000 UTC m=+252.231306873 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : secret "metrics-daemon-secret" not found Jul 31 14:14:52.846443 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:52.846402 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" podUID="b85ca420-fd83-4ed0-90b6-609395bc209c" Jul 31 14:14:52.852727 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:52.852708 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" Jul 31 14:14:52.863746 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:52.863691 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nwlqc" podUID="697d6e95-2340-45cf-8c34-626bd16dbd73" Jul 31 14:14:53.434998 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:53.434968 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:14:53.435168 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:53.434974 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nwlqc" Jul 31 14:14:53.435168 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:53.434974 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:14:53.998090 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:53.998052 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:14:57.762166 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:57.762118 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:14:57.762637 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:57.762195 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:14:57.762637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:57.762272 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 31 14:14:57.762637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:57.762291 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 31 14:14:57.762637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:57.762339 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert podName:b85ca420-fd83-4ed0-90b6-609395bc209c nodeName:}" failed. No retries permitted until 2026-07-31 14:16:59.762324518 +0000 UTC m=+285.270703562 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-xwfbh" (UID: "b85ca420-fd83-4ed0-90b6-609395bc209c") : secret "networking-console-plugin-cert" not found Jul 31 14:14:57.762637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:57.762292 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d79fc885d-lrzsb: secret "image-registry-tls" not found Jul 31 14:14:57.762637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:57.762438 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls podName:5e6c2de2-4244-4337-88b5-80176d3b4681 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:59.762424304 +0000 UTC m=+285.270803349 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls") pod "image-registry-7d79fc885d-lrzsb" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681") : secret "image-registry-tls" not found Jul 31 14:14:57.862649 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:14:57.862624 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:14:57.862776 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:57.862756 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 31 14:14:57.862817 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:14:57.862811 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls podName:697d6e95-2340-45cf-8c34-626bd16dbd73 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:59.86279816 +0000 UTC m=+285.371177206 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls") pod "dns-default-nwlqc" (UID: "697d6e95-2340-45cf-8c34-626bd16dbd73") : secret "dns-default-metrics-tls" not found Jul 31 14:15:07.977508 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:15:07.977418 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:15:15.201676 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:15:15.201619 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" podUID="7eae1fb9-92c1-4ab0-a455-73811336e4b2" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused" Jul 31 14:15:15.487216 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:15:15.487129 2558 generic.go:356] "Generic (PLEG): container finished" podID="7eae1fb9-92c1-4ab0-a455-73811336e4b2" containerID="844c9169ff8358316f7dd314ca86678ec8eb0036e1e8e80eef8b85dd5e550d10" exitCode=1 Jul 31 14:15:15.487216 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:15:15.487191 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" event={"ID":"7eae1fb9-92c1-4ab0-a455-73811336e4b2","Type":"ContainerDied","Data":"844c9169ff8358316f7dd314ca86678ec8eb0036e1e8e80eef8b85dd5e550d10"} Jul 31 14:15:15.487485 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:15:15.487471 2558 scope.go:117] "RemoveContainer" containerID="844c9169ff8358316f7dd314ca86678ec8eb0036e1e8e80eef8b85dd5e550d10" Jul 31 14:15:16.491830 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:15:16.491797 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" event={"ID":"7eae1fb9-92c1-4ab0-a455-73811336e4b2","Type":"ContainerStarted","Data":"566b639cb2b11595c42ae928da2b1a7dbd493638a17a67b89ed68a7363b954bb"} Jul 31 14:15:16.492288 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:15:16.492094 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:15:16.492715 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:15:16.492696 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:16:04.399414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:04.399374 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:26.804217 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:26.804158 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:16:26.804657 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:26.804308 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 31 14:16:26.804657 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:26.804383 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs podName:26920aff-943f-4177-8777-d186d5e99fc3 nodeName:}" failed. No retries permitted until 2026-07-31 14:18:28.80436338 +0000 UTC m=+374.312742451 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs") pod "network-metrics-daemon-kdtjx" (UID: "26920aff-943f-4177-8777-d186d5e99fc3") : secret "metrics-daemon-secret" not found Jul 31 14:16:29.775919 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.775881 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-776f6c5f98-h4thz"] Jul 31 14:16:29.778014 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.777993 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.783125 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.783098 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 31 14:16:29.783256 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.783231 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jj24m\"" Jul 31 14:16:29.783324 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.783229 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 31 14:16:29.785527 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.785509 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 31 14:16:29.785772 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.785628 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 31 14:16:29.785772 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.785649 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 31 14:16:29.786683 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.786668 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 31 14:16:29.796761 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.796731 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-776f6c5f98-h4thz"] Jul 31 14:16:29.827955 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.827892 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddwt\" (UniqueName: \"kubernetes.io/projected/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-kube-api-access-fddwt\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.827955 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.827950 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-stats-auth\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.828166 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.827984 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-default-certificate\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.828166 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.828077 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.828166 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.828094 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.929163 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.929125 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.929163 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.929160 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.929434 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.929190 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddwt\" (UniqueName: \"kubernetes.io/projected/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-kube-api-access-fddwt\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.929434 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.929212 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-stats-auth\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.929434 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.929231 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-default-certificate\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.929434 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:29.929310 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:16:29.929434 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:29.929329 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:30.429307252 +0000 UTC m=+255.937686319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : configmap references non-existent config key: service-ca.crt Jul 31 14:16:29.929434 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:29.929371 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:30.429354342 +0000 UTC m=+255.937733389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : secret "router-metrics-certs-default" not found Jul 31 14:16:29.931739 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.931711 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-stats-auth\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.931845 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.931741 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-default-certificate\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:29.942252 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:29.942227 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddwt\" (UniqueName: \"kubernetes.io/projected/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-kube-api-access-fddwt\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:30.010793 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:30.010723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:30.432737 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:30.432693 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:30.432737 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:30.432733 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:30.432984 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:30.432831 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:16:30.432984 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:30.432866 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:31.43284805 +0000 UTC m=+256.941227094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : configmap references non-existent config key: service-ca.crt Jul 31 14:16:30.432984 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:30.432890 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:31.432882032 +0000 UTC m=+256.941261075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : secret "router-metrics-certs-default" not found Jul 31 14:16:31.442087 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:31.442046 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:31.442087 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:31.442084 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:31.442504 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:31.442219 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:16:31.442504 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:31.442228 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:33.442210896 +0000 UTC m=+258.950589940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : configmap references non-existent config key: service-ca.crt Jul 31 14:16:31.442504 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:31.442271 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:33.442257833 +0000 UTC m=+258.950636877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : secret "router-metrics-certs-default" not found Jul 31 14:16:32.647205 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:32.647169 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:33.457610 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:33.457571 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:33.457610 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:33.457607 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:33.457824 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:33.457737 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:16:33.457824 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:33.457771 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:37.457752993 +0000 UTC m=+262.966132057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : configmap references non-existent config key: service-ca.crt Jul 31 14:16:33.457824 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:33.457796 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:37.457787432 +0000 UTC m=+262.966166475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : secret "router-metrics-certs-default" not found Jul 31 14:16:34.177116 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:34.177082 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:34.246479 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:34.246434 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:34.834107 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:34.834070 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:35.164155 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.164117 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l5rch"] Jul 31 14:16:35.166307 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.166286 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.169108 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.169075 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 31 14:16:35.169233 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.169124 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t7s2z\"" Jul 31 14:16:35.170468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.170452 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 31 14:16:35.170568 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.170489 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 31 14:16:35.170568 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.170514 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 31 14:16:35.177176 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.177150 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l5rch"] Jul 31 14:16:35.271566 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.271526 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.271744 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.271585 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.271744 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.271607 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc3215d1-396d-4cd5-973c-9332b7493c42-data-volume\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.271744 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.271734 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bcq5\" (UniqueName: \"kubernetes.io/projected/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-api-access-6bcq5\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.271860 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.271767 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc3215d1-396d-4cd5-973c-9332b7493c42-crio-socket\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.372383 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.372340 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bcq5\" (UniqueName: \"kubernetes.io/projected/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-api-access-6bcq5\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.372383 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.372389 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc3215d1-396d-4cd5-973c-9332b7493c42-crio-socket\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.372637 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.372420 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.372637 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.372451 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.372637 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.372477 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc3215d1-396d-4cd5-973c-9332b7493c42-data-volume\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.372637 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.372493 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc3215d1-396d-4cd5-973c-9332b7493c42-crio-socket\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.372637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:35.372549 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:35.372637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:35.372581 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:35.372637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:35.372600 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:35.872584298 +0000 UTC m=+261.380963347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:35.372637 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:35.372637 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:35.872623701 +0000 UTC m=+261.381002744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : configmap "kube-rbac-proxy" not found Jul 31 14:16:35.372997 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.372869 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc3215d1-396d-4cd5-973c-9332b7493c42-data-volume\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.382595 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.382579 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bcq5\" (UniqueName: \"kubernetes.io/projected/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-api-access-6bcq5\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.877449 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.877412 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.877616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.877454 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:35.877616 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:35.877521 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:35.877616 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:35.877551 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:35.877616 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:35.877585 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:36.877571848 +0000 UTC m=+262.385950892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : configmap "kube-rbac-proxy" not found Jul 31 14:16:35.877616 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:35.877611 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:36.877598253 +0000 UTC m=+262.385977296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:35.984958 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:35.984931 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:36.086851 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:36.086811 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:36.473866 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:36.473833 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:36.521818 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:36.521784 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:36.885749 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:36.885719 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:36.885940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:36.885764 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:36.885940 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:36.885877 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:36.886024 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:36.885964 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:38.885948655 +0000 UTC m=+264.394327699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:36.886024 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:36.885880 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:36.886092 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:36.886032 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:38.886020312 +0000 UTC m=+264.394399356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : configmap "kube-rbac-proxy" not found Jul 31 14:16:36.939660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:36.939635 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8wjhs_809b2748-9aad-4e40-864e-2252739a4f03/dns-node-resolver/0.log" Jul 31 14:16:37.373884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:37.373850 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:37.491363 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:37.491280 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:37.491363 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:37.491347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:37.491750 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:37.491446 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:45.491432257 +0000 UTC m=+270.999811301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : configmap references non-existent config key: service-ca.crt Jul 31 14:16:37.491750 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:37.491448 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 31 14:16:37.491750 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:37.491499 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs podName:6bc4182d-9581-47d4-bf1a-3e57ee04ffcb nodeName:}" failed. No retries permitted until 2026-07-31 14:16:45.491486165 +0000 UTC m=+270.999865214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs") pod "router-default-776f6c5f98-h4thz" (UID: "6bc4182d-9581-47d4-bf1a-3e57ee04ffcb") : secret "router-metrics-certs-default" not found Jul 31 14:16:38.138633 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.138606 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kd9wm_4c0cf086-f032-4b04-adcb-23df0705d67c/node-ca/0.log" Jul 31 14:16:38.161063 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.161029 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:38.263135 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.263099 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:38.364852 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.364814 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:38.466669 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.466582 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:38.659441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.659405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:38.760618 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.760506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:38.863943 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.863895 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:38.902251 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.902222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:38.902398 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.902263 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:38.902398 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:38.902346 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:38.902398 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:38.902381 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:38.902398 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:38.902397 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:42.902384704 +0000 UTC m=+268.410763752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : configmap "kube-rbac-proxy" not found Jul 31 14:16:38.902543 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:38.902452 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:42.902434552 +0000 UTC m=+268.410813596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:38.938879 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:38.938849 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bzp46_eabbfc47-226f-4ed0-a0aa-a495528d9f29/serve-healthcheck-canary/0.log" Jul 31 14:16:39.485949 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:39.485896 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:40.046340 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:40.046303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:40.147625 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:40.147586 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:41.612116 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:41.612083 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:41.712311 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:41.712264 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:41.905061 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:41.905026 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:42.049458 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:42.049428 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:42.248519 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:42.248428 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:42.699366 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:42.699332 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:42.937059 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:42.937027 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:42.937268 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:42.937078 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:42.937268 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:42.937195 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 31 14:16:42.937268 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:42.937261 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:50.937242087 +0000 UTC m=+276.445621147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : configmap "kube-rbac-proxy" not found Jul 31 14:16:42.937441 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:42.937197 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 31 14:16:42.937441 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:42.937340 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls podName:cc3215d1-396d-4cd5-973c-9332b7493c42 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:50.937324999 +0000 UTC m=+276.445704062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l5rch" (UID: "cc3215d1-396d-4cd5-973c-9332b7493c42") : secret "insights-runtime-extractor-tls" not found Jul 31 14:16:43.250343 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:43.250308 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:43.362556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:43.362513 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:43.655096 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:43.655061 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:44.057419 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:44.057335 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:44.253378 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:44.253341 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:44.654230 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:44.654201 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:44.705224 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:44.705175 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:45.048947 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:45.048845 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:45.558035 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:45.558001 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:45.558475 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:45.558050 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:45.558597 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:45.558581 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-service-ca-bundle\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:45.560424 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:45.560398 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc4182d-9581-47d4-bf1a-3e57ee04ffcb-metrics-certs\") pod \"router-default-776f6c5f98-h4thz\" (UID: \"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb\") " pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:45.687063 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:45.687021 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:45.804008 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:45.803975 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-776f6c5f98-h4thz"] Jul 31 14:16:45.806653 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:16:45.806620 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc4182d_9581_47d4_bf1a_3e57ee04ffcb.slice/crio-07170d6a79e086ee83b5629938c75ffc677c74f3f67785f08a2ebf17bde60808 WatchSource:0}: Error finding container 07170d6a79e086ee83b5629938c75ffc677c74f3f67785f08a2ebf17bde60808: Status 404 returned error can't find the container with id 07170d6a79e086ee83b5629938c75ffc677c74f3f67785f08a2ebf17bde60808 Jul 31 14:16:45.858561 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:45.858534 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:46.084636 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:46.084536 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:46.252433 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:46.252398 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:46.651113 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:46.651077 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:46.710670 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:46.710636 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-776f6c5f98-h4thz" event={"ID":"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb","Type":"ContainerStarted","Data":"f4fcf12efae133cba2156a113c91146a1bf8bf731cf50b49cf501643e97dc84b"} Jul 31 14:16:46.710823 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:46.710697 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-776f6c5f98-h4thz" event={"ID":"6bc4182d-9581-47d4-bf1a-3e57ee04ffcb","Type":"ContainerStarted","Data":"07170d6a79e086ee83b5629938c75ffc677c74f3f67785f08a2ebf17bde60808"} Jul 31 14:16:46.729368 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:46.729319 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-776f6c5f98-h4thz" podStartSLOduration=17.729305189 podStartE2EDuration="17.729305189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:46.728482348 +0000 UTC m=+272.236861414" watchObservedRunningTime="2026-07-31 14:16:46.729305189 +0000 UTC m=+272.237684255" Jul 31 14:16:47.257098 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:47.257055 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:47.400916 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:47.400868 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:47.657404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:47.657369 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:47.687836 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:47.687810 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:47.690344 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:47.690323 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:47.715261 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:47.715236 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:47.716374 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:47.716356 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-776f6c5f98-h4thz" Jul 31 14:16:47.975401 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:47.975308 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:48.253004 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:48.252919 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:48.503096 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:48.503005 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:48.647880 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:48.647848 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:48.699658 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:48.699624 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:48.854629 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:48.854539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:49.253341 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:49.253305 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:51.001757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:51.001724 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:51.001757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:51.001757 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:51.002221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:51.002204 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc3215d1-396d-4cd5-973c-9332b7493c42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:51.004022 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:51.004000 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc3215d1-396d-4cd5-973c-9332b7493c42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l5rch\" (UID: \"cc3215d1-396d-4cd5-973c-9332b7493c42\") " pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:51.075326 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:51.075294 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l5rch" Jul 31 14:16:51.192376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:51.192347 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l5rch"] Jul 31 14:16:51.195316 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:16:51.195286 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3215d1_396d_4cd5_973c_9332b7493c42.slice/crio-86c0a618dbbe50882a59235f2f67552037876ef5401b01e329401c072bc66fc6 WatchSource:0}: Error finding container 86c0a618dbbe50882a59235f2f67552037876ef5401b01e329401c072bc66fc6: Status 404 returned error can't find the container with id 86c0a618dbbe50882a59235f2f67552037876ef5401b01e329401c072bc66fc6 Jul 31 14:16:51.728355 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:51.728319 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l5rch" event={"ID":"cc3215d1-396d-4cd5-973c-9332b7493c42","Type":"ContainerStarted","Data":"77e0fd2975e34f4f8bf08e6019b0ef4610e17ed0bc4d2fa7d8332969a114e91b"} Jul 31 14:16:51.728503 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:51.728376 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l5rch" event={"ID":"cc3215d1-396d-4cd5-973c-9332b7493c42","Type":"ContainerStarted","Data":"86c0a618dbbe50882a59235f2f67552037876ef5401b01e329401c072bc66fc6"} Jul 31 14:16:53.735213 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:53.735180 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l5rch" event={"ID":"cc3215d1-396d-4cd5-973c-9332b7493c42","Type":"ContainerStarted","Data":"02f1bf4ed47a02b438fd214d09a446e92b9099b25cb07558e9ec6d1a6fbeb8bd"} Jul 31 14:16:54.741915 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:54.741864 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l5rch" event={"ID":"cc3215d1-396d-4cd5-973c-9332b7493c42","Type":"ContainerStarted","Data":"855e8bd8b5426db310fe671847983f571266d4101ca76b68021ae6861bc287fb"} Jul 31 14:16:54.760227 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:54.760185 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l5rch" podStartSLOduration=16.454444647 podStartE2EDuration="19.760149842s" totalImagesPullingTime="3.305705195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:16:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:54.758475914 +0000 UTC m=+280.266854979" watchObservedRunningTime="2026-07-31 14:16:54.760149842 +0000 UTC m=+280.268528940" Jul 31 14:16:56.435976 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:56.435920 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" podUID="b85ca420-fd83-4ed0-90b6-609395bc209c" Jul 31 14:16:56.436419 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:56.435977 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" Jul 31 14:16:56.436419 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:16:56.435977 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nwlqc" podUID="697d6e95-2340-45cf-8c34-626bd16dbd73" Jul 31 14:16:56.748697 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:56.748617 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:16:56.748867 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:56.748617 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:16:56.748867 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:56.748617 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nwlqc" Jul 31 14:16:56.864308 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:56.864275 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:56.912957 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:56.912924 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:58.609089 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.609056 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5"] Jul 31 14:16:58.612263 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.612249 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" Jul 31 14:16:58.615200 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.615182 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xcm69\"" Jul 31 14:16:58.616703 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.616682 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 31 14:16:58.623357 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.623335 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5"] Jul 31 14:16:58.686397 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.686362 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:58.759474 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.759445 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a56a565-de68-4e6b-ac45-d7421239d643-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-rk7w5\" (UID: \"1a56a565-de68-4e6b-ac45-d7421239d643\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" Jul 31 14:16:58.785008 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.784972 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:58.860730 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.860640 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a56a565-de68-4e6b-ac45-d7421239d643-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-rk7w5\" (UID: \"1a56a565-de68-4e6b-ac45-d7421239d643\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" Jul 31 14:16:58.863039 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.863014 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a56a565-de68-4e6b-ac45-d7421239d643-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-rk7w5\" (UID: \"1a56a565-de68-4e6b-ac45-d7421239d643\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" Jul 31 14:16:58.884309 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.884275 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:16:58.920434 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:58.920397 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" Jul 31 14:16:59.045493 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:59.045462 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5"] Jul 31 14:16:59.048893 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:16:59.048860 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a56a565_de68_4e6b_ac45_d7421239d643.slice/crio-f1ab4a91579750529ee9e58c14b6ef0f81ad73a97bfc2b1267390a92fe0419d3 WatchSource:0}: Error finding container f1ab4a91579750529ee9e58c14b6ef0f81ad73a97bfc2b1267390a92fe0419d3: Status 404 returned error can't find the container with id f1ab4a91579750529ee9e58c14b6ef0f81ad73a97bfc2b1267390a92fe0419d3 Jul 31 14:16:59.757971 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:59.757934 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" event={"ID":"1a56a565-de68-4e6b-ac45-d7421239d643","Type":"ContainerStarted","Data":"f1ab4a91579750529ee9e58c14b6ef0f81ad73a97bfc2b1267390a92fe0419d3"} Jul 31 14:16:59.767374 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:59.767342 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:16:59.767475 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:59.767386 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:16:59.769999 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:59.769975 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b85ca420-fd83-4ed0-90b6-609395bc209c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-xwfbh\" (UID: \"b85ca420-fd83-4ed0-90b6-609395bc209c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:16:59.770099 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:59.769975 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"image-registry-7d79fc885d-lrzsb\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:16:59.868279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:59.868245 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:16:59.870491 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:16:59.870469 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/697d6e95-2340-45cf-8c34-626bd16dbd73-metrics-tls\") pod \"dns-default-nwlqc\" (UID: \"697d6e95-2340-45cf-8c34-626bd16dbd73\") " pod="openshift-dns/dns-default-nwlqc" Jul 31 14:17:00.053644 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.053549 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9w89q\"" Jul 31 14:17:00.053644 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.053550 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9vdf\"" Jul 31 14:17:00.053644 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.053599 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jrsss\"" Jul 31 14:17:00.060028 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.060011 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:17:00.060119 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.060105 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" Jul 31 14:17:00.060166 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.060152 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nwlqc" Jul 31 14:17:00.204630 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.204600 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d79fc885d-lrzsb"] Jul 31 14:17:00.210387 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:00.210360 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6c2de2_4244_4337_88b5_80176d3b4681.slice/crio-e7d41160283c77198cfb1b9280b566e086a8049aee4fa62d4a0ca775fe2f8a32 WatchSource:0}: Error finding container e7d41160283c77198cfb1b9280b566e086a8049aee4fa62d4a0ca775fe2f8a32: Status 404 returned error can't find the container with id e7d41160283c77198cfb1b9280b566e086a8049aee4fa62d4a0ca775fe2f8a32 Jul 31 14:17:00.220062 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.220043 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh"] Jul 31 14:17:00.222990 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:00.222965 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85ca420_fd83_4ed0_90b6_609395bc209c.slice/crio-86f7ac3a5724e744368bf80378d809ad1861c350ce69ac536ef9b7742aae7705 WatchSource:0}: Error finding container 86f7ac3a5724e744368bf80378d809ad1861c350ce69ac536ef9b7742aae7705: Status 404 returned error can't find the container with id 86f7ac3a5724e744368bf80378d809ad1861c350ce69ac536ef9b7742aae7705 Jul 31 14:17:00.236279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.236257 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nwlqc"] Jul 31 14:17:00.244839 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:00.244818 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697d6e95_2340_45cf_8c34_626bd16dbd73.slice/crio-d595be602af0fab642b5c7ac80f238feb878cbecc0821a2cb6911af21c14b54b WatchSource:0}: Error finding container d595be602af0fab642b5c7ac80f238feb878cbecc0821a2cb6911af21c14b54b: Status 404 returned error can't find the container with id d595be602af0fab642b5c7ac80f238feb878cbecc0821a2cb6911af21c14b54b Jul 31 14:17:00.762193 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.762159 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwlqc" event={"ID":"697d6e95-2340-45cf-8c34-626bd16dbd73","Type":"ContainerStarted","Data":"d595be602af0fab642b5c7ac80f238feb878cbecc0821a2cb6911af21c14b54b"} Jul 31 14:17:00.763261 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.763226 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" event={"ID":"b85ca420-fd83-4ed0-90b6-609395bc209c","Type":"ContainerStarted","Data":"86f7ac3a5724e744368bf80378d809ad1861c350ce69ac536ef9b7742aae7705"} Jul 31 14:17:00.764584 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.764553 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" event={"ID":"5e6c2de2-4244-4337-88b5-80176d3b4681","Type":"ContainerStarted","Data":"81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff"} Jul 31 14:17:00.764673 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.764593 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" event={"ID":"5e6c2de2-4244-4337-88b5-80176d3b4681","Type":"ContainerStarted","Data":"e7d41160283c77198cfb1b9280b566e086a8049aee4fa62d4a0ca775fe2f8a32"} Jul 31 14:17:00.764738 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.764695 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:17:00.785535 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:00.785475 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" podStartSLOduration=285.785458215 podStartE2EDuration="4m45.785458215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:00.783540939 +0000 UTC m=+286.291920006" watchObservedRunningTime="2026-07-31 14:17:00.785458215 +0000 UTC m=+286.293837282" Jul 31 14:17:01.768936 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:01.768887 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" event={"ID":"b85ca420-fd83-4ed0-90b6-609395bc209c","Type":"ContainerStarted","Data":"485ddccee89b393ce63b682a21d14cbdc803174520253dd9e9ad3c571d3c0f1e"} Jul 31 14:17:01.870415 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:01.870381 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:01.929277 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:01.929228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:02.773864 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.773827 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwlqc" event={"ID":"697d6e95-2340-45cf-8c34-626bd16dbd73","Type":"ContainerStarted","Data":"b0963d00562bc343ae9bc97221637a108839730d729d1683a0a5e49906153b7f"} Jul 31 14:17:02.774341 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.773919 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwlqc" event={"ID":"697d6e95-2340-45cf-8c34-626bd16dbd73","Type":"ContainerStarted","Data":"e172d32d2d86cf748dbddd81b7e840d943097192ebc242fde14799e5a1f3590e"} Jul 31 14:17:02.774341 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.773948 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nwlqc" Jul 31 14:17:02.775335 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.775303 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" event={"ID":"1a56a565-de68-4e6b-ac45-d7421239d643","Type":"ContainerStarted","Data":"f90d64e49d01f2ff532e15d5206a222b870a813d86581e9de21a1312ab1a32ab"} Jul 31 14:17:02.775590 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.775566 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" Jul 31 14:17:02.780094 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.780074 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" Jul 31 14:17:02.791780 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.791739 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nwlqc" podStartSLOduration=251.775764671 podStartE2EDuration="4m13.791724157s" totalImagesPullingTime="2.015959486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:02.791475971 +0000 UTC m=+288.299855037" watchObservedRunningTime="2026-07-31 14:17:02.791724157 +0000 UTC m=+288.300103224" Jul 31 14:17:02.792084 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.792061 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-xwfbh" podStartSLOduration=263.581343476 podStartE2EDuration="4m24.792056213s" totalImagesPullingTime="1.210712737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:01.785588733 +0000 UTC m=+287.293967803" watchObservedRunningTime="2026-07-31 14:17:02.792056213 +0000 UTC m=+288.300435286" Jul 31 14:17:02.805560 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:02.805521 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-rk7w5" podStartSLOduration=2.101482463 podStartE2EDuration="4.805507193s" totalImagesPullingTime="2.70402473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:02.805452901 +0000 UTC m=+288.313832176" watchObservedRunningTime="2026-07-31 14:17:02.805507193 +0000 UTC m=+288.313886258" Jul 31 14:17:03.679656 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.679617 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:03.716672 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.716641 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-9t8h4"] Jul 31 14:17:03.719723 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.719708 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.722569 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.722550 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 31 14:17:03.724010 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.723988 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 31 14:17:03.724104 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.724084 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 31 14:17:03.724162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.724084 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2c9gj\"" Jul 31 14:17:03.724162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.724122 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 31 14:17:03.724433 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.724418 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 31 14:17:03.728468 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.728349 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-9t8h4"] Jul 31 14:17:03.747308 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.747282 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:03.799027 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.798995 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21efdddc-f93a-4975-8a95-dc2ac28811d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.799400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.799168 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21efdddc-f93a-4975-8a95-dc2ac28811d9-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.799400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.799212 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shphm\" (UniqueName: \"kubernetes.io/projected/21efdddc-f93a-4975-8a95-dc2ac28811d9-kube-api-access-shphm\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.799400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.799326 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21efdddc-f93a-4975-8a95-dc2ac28811d9-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.899980 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.899937 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21efdddc-f93a-4975-8a95-dc2ac28811d9-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.899980 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.899997 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21efdddc-f93a-4975-8a95-dc2ac28811d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.900204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.900046 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21efdddc-f93a-4975-8a95-dc2ac28811d9-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.900204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.900072 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shphm\" (UniqueName: \"kubernetes.io/projected/21efdddc-f93a-4975-8a95-dc2ac28811d9-kube-api-access-shphm\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.900608 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.900588 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21efdddc-f93a-4975-8a95-dc2ac28811d9-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.902374 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.902352 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21efdddc-f93a-4975-8a95-dc2ac28811d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.902466 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.902439 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/21efdddc-f93a-4975-8a95-dc2ac28811d9-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.908473 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.908441 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shphm\" (UniqueName: \"kubernetes.io/projected/21efdddc-f93a-4975-8a95-dc2ac28811d9-kube-api-access-shphm\") pod \"prometheus-operator-758d7cb874-9t8h4\" (UID: \"21efdddc-f93a-4975-8a95-dc2ac28811d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:03.913331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:03.913289 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:04.028605 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:04.028520 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" Jul 31 14:17:04.149863 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:04.149637 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-9t8h4"] Jul 31 14:17:04.151619 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:04.151594 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21efdddc_f93a_4975_8a95_dc2ac28811d9.slice/crio-cbb2890262b422dbb64164781f003e9166df4903a0d2315d0bad94c127466e33 WatchSource:0}: Error finding container cbb2890262b422dbb64164781f003e9166df4903a0d2315d0bad94c127466e33: Status 404 returned error can't find the container with id cbb2890262b422dbb64164781f003e9166df4903a0d2315d0bad94c127466e33 Jul 31 14:17:04.783506 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:04.783467 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" event={"ID":"21efdddc-f93a-4975-8a95-dc2ac28811d9","Type":"ContainerStarted","Data":"cbb2890262b422dbb64164781f003e9166df4903a0d2315d0bad94c127466e33"} Jul 31 14:17:05.750722 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:05.750681 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:05.789341 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:05.789260 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" event={"ID":"21efdddc-f93a-4975-8a95-dc2ac28811d9","Type":"ContainerStarted","Data":"11de7b6b712dee9a4557f0efc2ee8b6b333d9db982f037d393dac4433757b795"} Jul 31 14:17:05.789628 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:05.789607 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" event={"ID":"21efdddc-f93a-4975-8a95-dc2ac28811d9","Type":"ContainerStarted","Data":"e0900f645a2b361a635c45942aa667299fba0f2b322300a4490b192cf1c369e5"} Jul 31 14:17:05.807094 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:05.807049 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-9t8h4" podStartSLOduration=1.4559031 podStartE2EDuration="2.80703721s" totalImagesPullingTime="1.35113411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:05.805565646 +0000 UTC m=+291.313944712" watchObservedRunningTime="2026-07-31 14:17:05.80703721 +0000 UTC m=+291.315416276" Jul 31 14:17:07.832375 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:07.832342 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:08.053564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.053529 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:08.085401 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.085318 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf"] Jul 31 14:17:08.088595 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.088566 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.089484 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.089458 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2kfn4"] Jul 31 14:17:08.091836 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.091815 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 31 14:17:08.091956 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.091825 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 31 14:17:08.092457 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.092439 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.093250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.093228 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gk6c7\"" Jul 31 14:17:08.093336 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.093263 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 31 14:17:08.095109 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.095087 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 31 14:17:08.095202 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.095184 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zn2sg\"" Jul 31 14:17:08.095275 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.095233 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 31 14:17:08.095326 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.095298 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 31 14:17:08.099080 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.099050 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf"] Jul 31 14:17:08.123557 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.123522 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:08.128446 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128419 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.128552 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128452 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-sys\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.128552 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128475 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bzj\" (UniqueName: \"kubernetes.io/projected/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-api-access-b2bzj\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.128552 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128542 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.128733 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128570 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-metrics-client-ca\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.128733 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128634 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-textfile\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.128733 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128669 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.128733 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128704 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-tls\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.128855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128753 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-root\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.128855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128780 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6078041b-bb5e-4da7-ba12-f9224586efdf-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.128855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128804 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-wtmp\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.128855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128837 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.129013 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128863 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6078041b-bb5e-4da7-ba12-f9224586efdf-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.129013 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128891 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5mz\" (UniqueName: \"kubernetes.io/projected/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-kube-api-access-cv5mz\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.129013 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.128929 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.230066 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230024 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.230066 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230061 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-sys\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230089 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bzj\" (UniqueName: \"kubernetes.io/projected/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-api-access-b2bzj\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.230321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230155 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-sys\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230211 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230256 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-metrics-client-ca\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-textfile\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230575 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230334 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230575 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230382 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-tls\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230575 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230422 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-root\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230575 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230449 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6078041b-bb5e-4da7-ba12-f9224586efdf-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.230575 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230473 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-wtmp\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230575 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230514 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-root\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230575 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230523 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.230954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230577 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6078041b-bb5e-4da7-ba12-f9224586efdf-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.230954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230624 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5mz\" (UniqueName: \"kubernetes.io/projected/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-kube-api-access-cv5mz\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230652 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.230954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230763 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-textfile\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230851 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.230954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230941 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-wtmp\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.231247 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.230969 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-metrics-client-ca\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.232016 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.231458 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.232016 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.231814 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6078041b-bb5e-4da7-ba12-f9224586efdf-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.232208 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.232055 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6078041b-bb5e-4da7-ba12-f9224586efdf-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.233039 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.233019 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.233234 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.233215 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.233283 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.233258 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.233611 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.233593 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-node-exporter-tls\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.237889 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.237863 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bzj\" (UniqueName: \"kubernetes.io/projected/6078041b-bb5e-4da7-ba12-f9224586efdf-kube-api-access-b2bzj\") pod \"kube-state-metrics-6bd9664c4-nvwtf\" (UID: \"6078041b-bb5e-4da7-ba12-f9224586efdf\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.237964 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.237934 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5mz\" (UniqueName: \"kubernetes.io/projected/eae9b491-dfd9-4415-9fca-a2c5b4e433ff-kube-api-access-cv5mz\") pod \"node-exporter-2kfn4\" (UID: \"eae9b491-dfd9-4415-9fca-a2c5b4e433ff\") " pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.399819 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.399782 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" Jul 31 14:17:08.405259 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.405237 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2kfn4" Jul 31 14:17:08.416594 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:08.416562 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae9b491_dfd9_4415_9fca_a2c5b4e433ff.slice/crio-e3ee472a82bd843dadeacc739a3ae52e93d002f9e6d90cd1311cbd935308d4fd WatchSource:0}: Error finding container e3ee472a82bd843dadeacc739a3ae52e93d002f9e6d90cd1311cbd935308d4fd: Status 404 returned error can't find the container with id e3ee472a82bd843dadeacc739a3ae52e93d002f9e6d90cd1311cbd935308d4fd Jul 31 14:17:08.527372 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.527347 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf"] Jul 31 14:17:08.529469 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:08.529434 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6078041b_bb5e_4da7_ba12_f9224586efdf.slice/crio-2fcf718cfb195f47eafd9dd716949aa6ff6159327ea899f93aca84a71552e28e WatchSource:0}: Error finding container 2fcf718cfb195f47eafd9dd716949aa6ff6159327ea899f93aca84a71552e28e: Status 404 returned error can't find the container with id 2fcf718cfb195f47eafd9dd716949aa6ff6159327ea899f93aca84a71552e28e Jul 31 14:17:08.801306 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.801217 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" event={"ID":"6078041b-bb5e-4da7-ba12-f9224586efdf","Type":"ContainerStarted","Data":"2fcf718cfb195f47eafd9dd716949aa6ff6159327ea899f93aca84a71552e28e"} Jul 31 14:17:08.802236 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:08.802213 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2kfn4" event={"ID":"eae9b491-dfd9-4415-9fca-a2c5b4e433ff","Type":"ContainerStarted","Data":"e3ee472a82bd843dadeacc739a3ae52e93d002f9e6d90cd1311cbd935308d4fd"} Jul 31 14:17:09.138254 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.138219 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:17:09.145643 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.145613 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.148660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.148518 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 31 14:17:09.148660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.148534 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 31 14:17:09.148660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.148543 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 31 14:17:09.148660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.148555 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 31 14:17:09.148660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.148596 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 31 14:17:09.148971 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.148890 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 31 14:17:09.148971 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.148898 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 31 14:17:09.148971 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.148925 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rqwcl\"" Jul 31 14:17:09.149142 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.149125 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 31 14:17:09.149195 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.149144 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 31 14:17:09.155490 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.155468 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:17:09.237891 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.237857 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238077 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.237923 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-web-config\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238077 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238007 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238077 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238310 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238094 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238310 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238151 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238310 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238180 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-config-out\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238310 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238210 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238310 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238240 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238310 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238270 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238652 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238329 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238652 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238382 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfdg\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-kube-api-access-tlfdg\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.238652 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.238428 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339462 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339423 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339662 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339471 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339662 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339524 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-web-config\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339662 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339574 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339662 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339601 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339662 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339637 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339675 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339954 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:09.339789 2558 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 31 14:17:09.339954 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:09.339868 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls podName:5f25afae-727e-4702-9fec-e7d839578cc1 nodeName:}" failed. No retries permitted until 2026-07-31 14:17:09.839846783 +0000 UTC m=+295.348225830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1") : secret "alertmanager-main-tls" not found Jul 31 14:17:09.339954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339879 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.339954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339943 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-config-out\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.340217 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.339980 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.340217 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.340015 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.340217 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.340067 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.340217 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.340107 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.340217 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.340148 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfdg\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-kube-api-access-tlfdg\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.341013 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.340699 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.341143 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.341055 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.344664 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.344614 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-web-config\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.344821 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.344794 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.344959 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.344942 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.345039 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.344964 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-config-out\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.345376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.345332 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.345735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.345711 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.346199 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.346176 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.346422 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.346400 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.348587 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.348566 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfdg\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-kube-api-access-tlfdg\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.458637 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.458507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:09.509308 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.509263 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:09.696983 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.696940 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:09.844657 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.844569 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:09.847522 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:09.847493 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:10.055994 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.055958 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:17:10.234816 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.234786 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:17:10.235175 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:10.234855 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f25afae_727e_4702_9fec_e7d839578cc1.slice/crio-7cdefd4ef0e0f6e42424560e4ed3e9a4577dde68d52374e49815b95b3ee670a7 WatchSource:0}: Error finding container 7cdefd4ef0e0f6e42424560e4ed3e9a4577dde68d52374e49815b95b3ee670a7: Status 404 returned error can't find the container with id 7cdefd4ef0e0f6e42424560e4ed3e9a4577dde68d52374e49815b95b3ee670a7 Jul 31 14:17:10.812451 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.812246 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" event={"ID":"6078041b-bb5e-4da7-ba12-f9224586efdf","Type":"ContainerStarted","Data":"40b558c7c083abd3f94e5d1171cfd2320b9125dcb28b8625327f49b978a8afcc"} Jul 31 14:17:10.812451 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.812319 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" event={"ID":"6078041b-bb5e-4da7-ba12-f9224586efdf","Type":"ContainerStarted","Data":"5d3bc85b0403189ce14f6be5dbe3bc8599822d08afcdc17f3e3b43d63b63c53e"} Jul 31 14:17:10.812451 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.812352 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" event={"ID":"6078041b-bb5e-4da7-ba12-f9224586efdf","Type":"ContainerStarted","Data":"964cccee55991c1449dadb1b64552cbf29544ea16026ef9303a83f61227d59fe"} Jul 31 14:17:10.813725 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.813694 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerStarted","Data":"7cdefd4ef0e0f6e42424560e4ed3e9a4577dde68d52374e49815b95b3ee670a7"} Jul 31 14:17:10.815447 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.815421 2558 generic.go:356] "Generic (PLEG): container finished" podID="eae9b491-dfd9-4415-9fca-a2c5b4e433ff" containerID="edde7f4eb952ddb5e9fe41ad0318cffee596e8d1447dacc59dac8a6f42629a8b" exitCode=0 Jul 31 14:17:10.815573 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.815450 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2kfn4" event={"ID":"eae9b491-dfd9-4415-9fca-a2c5b4e433ff","Type":"ContainerDied","Data":"edde7f4eb952ddb5e9fe41ad0318cffee596e8d1447dacc59dac8a6f42629a8b"} Jul 31 14:17:10.839624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.839568 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-nvwtf" podStartSLOduration=1.445009861 podStartE2EDuration="2.839549096s" totalImagesPullingTime="1.394539235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:10.837471666 +0000 UTC m=+296.345850734" watchObservedRunningTime="2026-07-31 14:17:10.839549096 +0000 UTC m=+296.347928164" Jul 31 14:17:10.978095 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:10.978045 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kdtjx" podUID="26920aff-943f-4177-8777-d186d5e99fc3" Jul 31 14:17:10.993870 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:10.993823 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:11.046417 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:11.046378 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:11.252532 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:11.252494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:11.703352 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:11.703308 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:11.822625 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:11.822586 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2kfn4" event={"ID":"eae9b491-dfd9-4415-9fca-a2c5b4e433ff","Type":"ContainerStarted","Data":"80a7418871a588ef41503230f169eaaedd7a430e20d1d332cfafbb9ba506ff7a"} Jul 31 14:17:11.822804 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:11.822644 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2kfn4" event={"ID":"eae9b491-dfd9-4415-9fca-a2c5b4e433ff","Type":"ContainerStarted","Data":"b84234cd66775de976e3cf056e9bdae9f51987800f6ae1a0a59748dfb670a4a1"} Jul 31 14:17:11.828683 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:11.828651 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f25afae-727e-4702-9fec-e7d839578cc1" containerID="03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378" exitCode=0 Jul 31 14:17:11.828831 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:11.828727 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerDied","Data":"03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378"} Jul 31 14:17:11.843244 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:11.843201 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2kfn4" podStartSLOduration=2.250125758 podStartE2EDuration="3.843187444s" totalImagesPullingTime="1.505339842s" totalInitContainerRuntime="87.721844ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:11.841328618 +0000 UTC m=+297.349707675" watchObservedRunningTime="2026-07-31 14:17:11.843187444 +0000 UTC m=+297.351566509" Jul 31 14:17:12.780008 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.779974 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nwlqc" Jul 31 14:17:12.857078 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.857039 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9"] Jul 31 14:17:12.860615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.860590 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:12.863253 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.863227 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 31 14:17:12.863371 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.863347 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c9lw8\"" Jul 31 14:17:12.868485 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.868431 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:12.869389 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.869322 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9"] Jul 31 14:17:12.974811 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.974781 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4514041-d510-4074-9f05-d8f68f222e95-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-f8dk9\" (UID: \"d4514041-d510-4074-9f05-d8f68f222e95\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:12.975029 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:12.974870 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zvr\" (UniqueName: \"kubernetes.io/projected/d4514041-d510-4074-9f05-d8f68f222e95-kube-api-access-w6zvr\") pod \"monitoring-plugin-6dffbbd67-f8dk9\" (UID: \"d4514041-d510-4074-9f05-d8f68f222e95\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:13.078255 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.078167 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4514041-d510-4074-9f05-d8f68f222e95-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-f8dk9\" (UID: \"d4514041-d510-4074-9f05-d8f68f222e95\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:13.078255 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.078255 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zvr\" (UniqueName: \"kubernetes.io/projected/d4514041-d510-4074-9f05-d8f68f222e95-kube-api-access-w6zvr\") pod \"monitoring-plugin-6dffbbd67-f8dk9\" (UID: \"d4514041-d510-4074-9f05-d8f68f222e95\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:13.078483 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:13.078331 2558 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 31 14:17:13.078483 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:13.078415 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4514041-d510-4074-9f05-d8f68f222e95-monitoring-plugin-cert podName:d4514041-d510-4074-9f05-d8f68f222e95 nodeName:}" failed. No retries permitted until 2026-07-31 14:17:13.578394115 +0000 UTC m=+299.086773171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d4514041-d510-4074-9f05-d8f68f222e95-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-f8dk9" (UID: "d4514041-d510-4074-9f05-d8f68f222e95") : secret "monitoring-plugin-cert" not found Jul 31 14:17:13.087429 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.087403 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zvr\" (UniqueName: \"kubernetes.io/projected/d4514041-d510-4074-9f05-d8f68f222e95-kube-api-access-w6zvr\") pod \"monitoring-plugin-6dffbbd67-f8dk9\" (UID: \"d4514041-d510-4074-9f05-d8f68f222e95\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:13.122412 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.122372 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:13.583006 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.582977 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4514041-d510-4074-9f05-d8f68f222e95-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-f8dk9\" (UID: \"d4514041-d510-4074-9f05-d8f68f222e95\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:13.585378 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.585354 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4514041-d510-4074-9f05-d8f68f222e95-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-f8dk9\" (UID: \"d4514041-d510-4074-9f05-d8f68f222e95\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:13.773098 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.773063 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:13.840328 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.840104 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerStarted","Data":"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99"} Jul 31 14:17:13.840328 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.840193 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerStarted","Data":"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2"} Jul 31 14:17:13.840328 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.840238 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerStarted","Data":"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac"} Jul 31 14:17:13.840328 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.840282 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerStarted","Data":"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc"} Jul 31 14:17:13.914648 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:13.914624 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9"] Jul 31 14:17:13.917033 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:13.917005 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4514041_d510_4074_9f05_d8f68f222e95.slice/crio-cf8e83a19f57cb9a0086eb8f9f313aa45c32e1d039b0581840dda482b1a13ead WatchSource:0}: Error finding container cf8e83a19f57cb9a0086eb8f9f313aa45c32e1d039b0581840dda482b1a13ead: Status 404 returned error can't find the container with id cf8e83a19f57cb9a0086eb8f9f313aa45c32e1d039b0581840dda482b1a13ead Jul 31 14:17:14.248198 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:14.248159 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:14.847394 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:14.847357 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerStarted","Data":"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6"} Jul 31 14:17:14.847867 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:14.847472 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerStarted","Data":"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228"} Jul 31 14:17:14.848577 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:14.848553 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" event={"ID":"d4514041-d510-4074-9f05-d8f68f222e95","Type":"ContainerStarted","Data":"cf8e83a19f57cb9a0086eb8f9f313aa45c32e1d039b0581840dda482b1a13ead"} Jul 31 14:17:14.875588 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:14.875523 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.22578618 podStartE2EDuration="5.875502234s" totalImagesPullingTime="3.633887487s" totalInitContainerRuntime="15.828567ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:14.874387276 +0000 UTC m=+300.382766343" watchObservedRunningTime="2026-07-31 14:17:14.875502234 +0000 UTC m=+300.383881301" Jul 31 14:17:15.207605 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.207583 2558 kubelet.go:1648] "Image garbage collection succeeded" Jul 31 14:17:15.853571 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.853482 2558 generic.go:356] "Generic (PLEG): container finished" podID="7eae1fb9-92c1-4ab0-a455-73811336e4b2" containerID="566b639cb2b11595c42ae928da2b1a7dbd493638a17a67b89ed68a7363b954bb" exitCode=1 Jul 31 14:17:15.854037 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.853557 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" event={"ID":"7eae1fb9-92c1-4ab0-a455-73811336e4b2","Type":"ContainerDied","Data":"566b639cb2b11595c42ae928da2b1a7dbd493638a17a67b89ed68a7363b954bb"} Jul 31 14:17:15.854037 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.853663 2558 scope.go:117] "RemoveContainer" containerID="844c9169ff8358316f7dd314ca86678ec8eb0036e1e8e80eef8b85dd5e550d10" Jul 31 14:17:15.854153 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.854093 2558 scope.go:117] "RemoveContainer" containerID="566b639cb2b11595c42ae928da2b1a7dbd493638a17a67b89ed68a7363b954bb" Jul 31 14:17:15.854313 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:15.854283 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-6968d6c77-cb4tv_open-cluster-management-agent-addon(7eae1fb9-92c1-4ab0-a455-73811336e4b2)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" podUID="7eae1fb9-92c1-4ab0-a455-73811336e4b2" Jul 31 14:17:15.855220 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.855189 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" event={"ID":"d4514041-d510-4074-9f05-d8f68f222e95","Type":"ContainerStarted","Data":"cd272ce57a2a1fbe771e31a0711da8380e0dba6d39a3a9a05d650ac069a121b2"} Jul 31 14:17:15.855565 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.855544 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:15.860327 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.860261 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" Jul 31 14:17:15.886586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:15.886538 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-f8dk9" podStartSLOduration=2.615458561 podStartE2EDuration="3.886526124s" totalImagesPullingTime="1.271067563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:15.885410164 +0000 UTC m=+301.393789225" watchObservedRunningTime="2026-07-31 14:17:15.886526124 +0000 UTC m=+301.394905215" Jul 31 14:17:16.492231 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:16.492181 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:17:16.860780 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:16.860706 2558 scope.go:117] "RemoveContainer" containerID="566b639cb2b11595c42ae928da2b1a7dbd493638a17a67b89ed68a7363b954bb" Jul 31 14:17:16.861131 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:16.860888 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-6968d6c77-cb4tv_open-cluster-management-agent-addon(7eae1fb9-92c1-4ab0-a455-73811336e4b2)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" podUID="7eae1fb9-92c1-4ab0-a455-73811336e4b2" Jul 31 14:17:17.116970 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:17.116933 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:17.174878 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:17.174842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:19.150200 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:19.150160 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:19.252513 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:19.252478 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:19.532024 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:19.531923 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:20.065036 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.065004 2558 patch_prober.go:28] interesting pod/image-registry-7d79fc885d-lrzsb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 31 14:17:20.065244 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.065060 2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 14:17:20.409512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.409476 2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:17:20.410034 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.409972 2558 scope.go:117] "RemoveContainer" containerID="566b639cb2b11595c42ae928da2b1a7dbd493638a17a67b89ed68a7363b954bb" Jul 31 14:17:20.410195 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:20.410171 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-6968d6c77-cb4tv_open-cluster-management-agent-addon(7eae1fb9-92c1-4ab0-a455-73811336e4b2)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" podUID="7eae1fb9-92c1-4ab0-a455-73811336e4b2" Jul 31 14:17:20.440579 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.440538 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d79fc885d-lrzsb"] Jul 31 14:17:20.445034 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.445008 2558 patch_prober.go:28] interesting pod/image-registry-7d79fc885d-lrzsb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 31 14:17:20.445178 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.445050 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 31 14:17:20.527276 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.527233 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:20.627645 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:20.627601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:21.888439 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:21.888396 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:21.988811 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:21.988775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:25.977124 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:25.977072 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:17:26.901671 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:26.901605 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:26.964646 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:26.964612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:28.596739 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:28.596703 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:28.698105 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:28.698064 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:28.747540 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:28.747507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:28.860089 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:28.860000 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:29.247933 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:29.247881 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:29.298174 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:29.298145 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:29.649079 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:29.649042 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:29.866860 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:29.866819 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:29.920464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:29.920383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:30.049734 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:30.049703 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:30.252474 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:30.252388 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:30.307292 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:30.307259 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:30.445634 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:30.445607 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:17:30.879530 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:30.879499 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:31.589694 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:31.589658 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:31.689937 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:31.689893 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:31.977460 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:31.977428 2558 scope.go:117] "RemoveContainer" containerID="566b639cb2b11595c42ae928da2b1a7dbd493638a17a67b89ed68a7363b954bb" Jul 31 14:17:32.866947 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:32.866888 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:32.918222 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:32.918190 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" event={"ID":"7eae1fb9-92c1-4ab0-a455-73811336e4b2","Type":"ContainerStarted","Data":"92b2e08d22d5695bb280a0a84e5248e4525dc71f3f213935ef6700b48e250729"} Jul 31 14:17:32.918522 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:32.918504 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:17:32.920127 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:32.920105 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6968d6c77-cb4tv" Jul 31 14:17:34.605033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:34.604992 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:40.659742 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:40.659709 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:40.715003 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:40.714960 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:41.861283 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:41.861242 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:41.913027 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:41.912997 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:45.459524 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.459459 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" containerName="registry" containerID="cri-o://81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff" gracePeriod=30 Jul 31 14:17:45.722537 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.722481 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:17:45.762615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.762586 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-image-registry-private-configuration\") pod \"5e6c2de2-4244-4337-88b5-80176d3b4681\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " Jul 31 14:17:45.762770 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.762623 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-bound-sa-token\") pod \"5e6c2de2-4244-4337-88b5-80176d3b4681\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " Jul 31 14:17:45.762770 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.762653 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnvzp\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-kube-api-access-tnvzp\") pod \"5e6c2de2-4244-4337-88b5-80176d3b4681\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " Jul 31 14:17:45.762770 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.762678 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-trusted-ca\") pod \"5e6c2de2-4244-4337-88b5-80176d3b4681\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " Jul 31 14:17:45.762770 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.762727 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-certificates\") pod \"5e6c2de2-4244-4337-88b5-80176d3b4681\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " Jul 31 14:17:45.762770 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.762755 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e6c2de2-4244-4337-88b5-80176d3b4681-ca-trust-extracted\") pod \"5e6c2de2-4244-4337-88b5-80176d3b4681\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " Jul 31 14:17:45.763018 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.762783 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") pod \"5e6c2de2-4244-4337-88b5-80176d3b4681\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " Jul 31 14:17:45.763018 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.762821 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-installation-pull-secrets\") pod \"5e6c2de2-4244-4337-88b5-80176d3b4681\" (UID: \"5e6c2de2-4244-4337-88b5-80176d3b4681\") " Jul 31 14:17:45.763346 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.763292 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5e6c2de2-4244-4337-88b5-80176d3b4681" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:17:45.763690 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.763669 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5e6c2de2-4244-4337-88b5-80176d3b4681" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:17:45.765540 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.765510 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5e6c2de2-4244-4337-88b5-80176d3b4681" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:17:45.765668 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.765569 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5e6c2de2-4244-4337-88b5-80176d3b4681" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:17:45.765668 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.765626 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-kube-api-access-tnvzp" (OuterVolumeSpecName: "kube-api-access-tnvzp") pod "5e6c2de2-4244-4337-88b5-80176d3b4681" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681"). InnerVolumeSpecName "kube-api-access-tnvzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:17:45.765668 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.765623 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5e6c2de2-4244-4337-88b5-80176d3b4681" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:17:45.765668 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.765658 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5e6c2de2-4244-4337-88b5-80176d3b4681" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:17:45.771598 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.771569 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6c2de2-4244-4337-88b5-80176d3b4681-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5e6c2de2-4244-4337-88b5-80176d3b4681" (UID: "5e6c2de2-4244-4337-88b5-80176d3b4681"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:17:45.864162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.864128 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnvzp\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-kube-api-access-tnvzp\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:17:45.864162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.864159 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-trusted-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:17:45.864162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.864171 2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-certificates\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:17:45.864411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.864184 2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e6c2de2-4244-4337-88b5-80176d3b4681-ca-trust-extracted\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:17:45.864411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.864195 2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-registry-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:17:45.864411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.864206 2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-installation-pull-secrets\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:17:45.864411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.864218 2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e6c2de2-4244-4337-88b5-80176d3b4681-image-registry-private-configuration\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:17:45.864411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.864230 2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e6c2de2-4244-4337-88b5-80176d3b4681-bound-sa-token\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:17:45.968818 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.968785 2558 generic.go:356] "Generic (PLEG): container finished" podID="5e6c2de2-4244-4337-88b5-80176d3b4681" containerID="81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff" exitCode=0 Jul 31 14:17:45.968995 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.968830 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" Jul 31 14:17:45.968995 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.968866 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" event={"ID":"5e6c2de2-4244-4337-88b5-80176d3b4681","Type":"ContainerDied","Data":"81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff"} Jul 31 14:17:45.968995 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.968931 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d79fc885d-lrzsb" event={"ID":"5e6c2de2-4244-4337-88b5-80176d3b4681","Type":"ContainerDied","Data":"e7d41160283c77198cfb1b9280b566e086a8049aee4fa62d4a0ca775fe2f8a32"} Jul 31 14:17:45.968995 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.968962 2558 scope.go:117] "RemoveContainer" containerID="81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff" Jul 31 14:17:45.977556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.977537 2558 scope.go:117] "RemoveContainer" containerID="81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff" Jul 31 14:17:45.977796 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:17:45.977774 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff\": container with ID starting with 81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff not found: ID does not exist" containerID="81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff" Jul 31 14:17:45.977842 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.977804 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff"} err="failed to get container status \"81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff\": rpc error: code = NotFound desc = could not find container \"81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff\": container with ID starting with 81672d12f678c1eef00c54993cc3fbea607f7a20b75237b6a5d72c41ce0563ff not found: ID does not exist" Jul 31 14:17:45.989314 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.989287 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d79fc885d-lrzsb"] Jul 31 14:17:45.992863 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:45.992843 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d79fc885d-lrzsb"] Jul 31 14:17:46.981732 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:46.981704 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" path="/var/lib/kubelet/pods/5e6c2de2-4244-4337-88b5-80176d3b4681/volumes" Jul 31 14:17:47.071614 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:47.071577 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:47.125223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:47.125187 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:48.639044 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:48.639013 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:48.738190 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:48.738158 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:49.268321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.268281 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-579565665b-wbl52"] Jul 31 14:17:49.268569 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.268557 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" containerName="registry" Jul 31 14:17:49.268613 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.268571 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" containerName="registry" Jul 31 14:17:49.268647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.268628 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6c2de2-4244-4337-88b5-80176d3b4681" containerName="registry" Jul 31 14:17:49.272148 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.272128 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.274773 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.274741 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 31 14:17:49.274969 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.274848 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 31 14:17:49.276110 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.276089 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 31 14:17:49.276110 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.276101 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bv27n\"" Jul 31 14:17:49.276314 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.276130 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 31 14:17:49.276314 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.276146 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 31 14:17:49.276314 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.276146 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 14:17:49.276314 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.276204 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 14:17:49.280983 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.280967 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 31 14:17:49.282176 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.282157 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579565665b-wbl52"] Jul 31 14:17:49.296504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.296475 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-service-ca\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.296627 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.296515 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-console-config\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.296627 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.296537 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-oauth-config\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.296627 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.296594 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhpb\" (UniqueName: \"kubernetes.io/projected/ee03d7a9-eeac-4784-ab80-2504a1177760-kube-api-access-zmhpb\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.296725 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.296649 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-serving-cert\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.296725 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.296670 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-trusted-ca-bundle\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.296725 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.296692 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-oauth-serving-cert\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.397186 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.397150 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhpb\" (UniqueName: \"kubernetes.io/projected/ee03d7a9-eeac-4784-ab80-2504a1177760-kube-api-access-zmhpb\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.397186 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.397194 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-serving-cert\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.397400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.397222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-trusted-ca-bundle\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.397400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.397257 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-oauth-serving-cert\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.397400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.397319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-service-ca\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.397400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.397360 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-console-config\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.397400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.397389 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-oauth-config\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.398082 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.398055 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-oauth-serving-cert\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.398228 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.398116 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-console-config\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.398306 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.398279 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-service-ca\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.398490 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.398470 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-trusted-ca-bundle\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.399696 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.399673 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-serving-cert\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.399920 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.399880 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-oauth-config\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.405449 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.405421 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhpb\" (UniqueName: \"kubernetes.io/projected/ee03d7a9-eeac-4784-ab80-2504a1177760-kube-api-access-zmhpb\") pod \"console-579565665b-wbl52\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.581380 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.581282 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:49.719265 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.719237 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579565665b-wbl52"] Jul 31 14:17:49.721823 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:17:49.721796 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee03d7a9_eeac_4784_ab80_2504a1177760.slice/crio-fd545b40522fcc944536f1d1e89ce79228f1ed5417c32d339ecc07ff18ac0538 WatchSource:0}: Error finding container fd545b40522fcc944536f1d1e89ce79228f1ed5417c32d339ecc07ff18ac0538: Status 404 returned error can't find the container with id fd545b40522fcc944536f1d1e89ce79228f1ed5417c32d339ecc07ff18ac0538 Jul 31 14:17:49.724051 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.724033 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:17:49.986917 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:49.986872 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579565665b-wbl52" event={"ID":"ee03d7a9-eeac-4784-ab80-2504a1177760","Type":"ContainerStarted","Data":"fd545b40522fcc944536f1d1e89ce79228f1ed5417c32d339ecc07ff18ac0538"} Jul 31 14:17:51.067835 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:51.067792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:51.119038 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:51.118997 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:53.074787 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:53.074727 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:53.137025 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:53.136987 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:55.007048 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:55.007012 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579565665b-wbl52" event={"ID":"ee03d7a9-eeac-4784-ab80-2504a1177760","Type":"ContainerStarted","Data":"25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef"} Jul 31 14:17:55.024351 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:55.024299 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-579565665b-wbl52" podStartSLOduration=1.510397121 podStartE2EDuration="6.024284571s" totalImagesPullingTime="4.51388745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:55.023499826 +0000 UTC m=+340.531878892" watchObservedRunningTime="2026-07-31 14:17:55.024284571 +0000 UTC m=+340.532663731" Jul 31 14:17:56.859304 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:56.859260 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:56.910433 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:56.910392 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:17:59.582157 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:59.582122 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:59.582554 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:59.582210 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:59.586737 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:59.586718 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-579565665b-wbl52" Jul 31 14:17:59.876491 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:17:59.876450 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:00.030526 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:00.030498 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-579565665b-wbl52" Jul 31 14:18:00.679882 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:00.679851 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:00.734537 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:00.734499 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:02.903501 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:02.903464 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:02.956074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:02.956039 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:03.277169 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:03.277082 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:03.326313 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:03.326279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:11.862540 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:11.862447 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:11.918790 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:11.918757 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:27.247262 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:27.247214 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:27.624432 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:27.624397 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:28.297107 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.297071 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:18:28.297507 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.297372 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="alertmanager" containerID="cri-o://cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc" gracePeriod=120 Jul 31 14:18:28.297507 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.297438 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy-metric" containerID="cri-o://1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228" gracePeriod=120 Jul 31 14:18:28.297507 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.297465 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="config-reloader" containerID="cri-o://b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac" gracePeriod=120 Jul 31 14:18:28.297507 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.297447 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy-web" containerID="cri-o://fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2" gracePeriod=120 Jul 31 14:18:28.297713 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.297490 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy" containerID="cri-o://5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99" gracePeriod=120 Jul 31 14:18:28.297713 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.297556 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="prom-label-proxy" containerID="cri-o://9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6" gracePeriod=120 Jul 31 14:18:28.841095 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.841057 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:18:28.843385 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.843363 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26920aff-943f-4177-8777-d186d5e99fc3-metrics-certs\") pod \"network-metrics-daemon-kdtjx\" (UID: \"26920aff-943f-4177-8777-d186d5e99fc3\") " pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:18:28.980346 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.980314 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lwbh\"" Jul 31 14:18:28.987838 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:28.987808 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdtjx" Jul 31 14:18:29.114976 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.114945 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kdtjx"] Jul 31 14:18:29.117442 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:18:29.117417 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26920aff_943f_4177_8777_d186d5e99fc3.slice/crio-72921d0ebf25a037f679a47e6821ca3634c76328e3f14cd9feb50412c172b3a3 WatchSource:0}: Error finding container 72921d0ebf25a037f679a47e6821ca3634c76328e3f14cd9feb50412c172b3a3: Status 404 returned error can't find the container with id 72921d0ebf25a037f679a47e6821ca3634c76328e3f14cd9feb50412c172b3a3 Jul 31 14:18:29.135803 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.135772 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdtjx" event={"ID":"26920aff-943f-4177-8777-d186d5e99fc3","Type":"ContainerStarted","Data":"72921d0ebf25a037f679a47e6821ca3634c76328e3f14cd9feb50412c172b3a3"} Jul 31 14:18:29.139048 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.139024 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f25afae-727e-4702-9fec-e7d839578cc1" containerID="9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6" exitCode=0 Jul 31 14:18:29.139048 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.139044 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f25afae-727e-4702-9fec-e7d839578cc1" containerID="5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99" exitCode=0 Jul 31 14:18:29.139181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.139051 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f25afae-727e-4702-9fec-e7d839578cc1" containerID="b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac" exitCode=0 Jul 31 14:18:29.139181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.139057 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f25afae-727e-4702-9fec-e7d839578cc1" containerID="cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc" exitCode=0 Jul 31 14:18:29.139181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.139100 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerDied","Data":"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6"} Jul 31 14:18:29.139181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.139158 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerDied","Data":"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99"} Jul 31 14:18:29.139301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.139186 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerDied","Data":"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac"} Jul 31 14:18:29.139301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.139214 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerDied","Data":"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc"} Jul 31 14:18:29.536361 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.536337 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:29.546030 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546008 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546161 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546035 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-cluster-tls-config\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546161 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546071 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-trusted-ca-bundle\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546161 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546097 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlfdg\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-kube-api-access-tlfdg\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546298 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546220 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-metrics-client-ca\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546298 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546289 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-tls-assets\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546330 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-main-db\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546354 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-config-volume\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546383 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546416 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-web-config\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546462 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546478 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:29.546556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546490 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546540 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-config-out\") pod \"5f25afae-727e-4702-9fec-e7d839578cc1\" (UID: \"5f25afae-727e-4702-9fec-e7d839578cc1\") " Jul 31 14:18:29.546786 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546633 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:18:29.546840 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546800 2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.546840 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.546821 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f25afae-727e-4702-9fec-e7d839578cc1-metrics-client-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.547436 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.547148 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:18:29.548941 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.548898 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:29.549450 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.549415 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-config-out" (OuterVolumeSpecName: "config-out") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:18:29.549552 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.549476 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-config-volume" (OuterVolumeSpecName: "config-volume") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:29.549552 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.549499 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:29.550044 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.550020 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:29.550162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.550126 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-kube-api-access-tlfdg" (OuterVolumeSpecName: "kube-api-access-tlfdg") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "kube-api-access-tlfdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:18:29.550225 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.550158 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:18:29.550523 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.550495 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:29.553502 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.553477 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:29.558879 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.558854 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-web-config" (OuterVolumeSpecName: "web-config") pod "5f25afae-727e-4702-9fec-e7d839578cc1" (UID: "5f25afae-727e-4702-9fec-e7d839578cc1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647473 2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-alertmanager-main-db\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647507 2558 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647518 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647528 2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-web-config\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647536 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647545 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-main-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647553 2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f25afae-727e-4702-9fec-e7d839578cc1-config-out\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647561 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647562 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647569 2558 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f25afae-727e-4702-9fec-e7d839578cc1-cluster-tls-config\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647976 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647579 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfdg\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-kube-api-access-tlfdg\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:29.647976 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:29.647588 2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f25afae-727e-4702-9fec-e7d839578cc1-tls-assets\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:18:30.146092 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.146058 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f25afae-727e-4702-9fec-e7d839578cc1" containerID="1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228" exitCode=0 Jul 31 14:18:30.146092 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.146085 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f25afae-727e-4702-9fec-e7d839578cc1" containerID="fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2" exitCode=0 Jul 31 14:18:30.146342 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.146133 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerDied","Data":"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228"} Jul 31 14:18:30.146342 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.146147 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.146342 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.146231 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerDied","Data":"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2"} Jul 31 14:18:30.146342 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.146260 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f25afae-727e-4702-9fec-e7d839578cc1","Type":"ContainerDied","Data":"7cdefd4ef0e0f6e42424560e4ed3e9a4577dde68d52374e49815b95b3ee670a7"} Jul 31 14:18:30.146342 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.146296 2558 scope.go:117] "RemoveContainer" containerID="9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6" Jul 31 14:18:30.170416 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.170388 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:18:30.175006 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.174980 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:18:30.176111 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.176094 2558 scope.go:117] "RemoveContainer" containerID="1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228" Jul 31 14:18:30.184136 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.184118 2558 scope.go:117] "RemoveContainer" containerID="5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99" Jul 31 14:18:30.191649 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.191624 2558 scope.go:117] "RemoveContainer" containerID="fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.198984 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199336 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy-web" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199353 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy-web" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199373 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199381 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199396 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="alertmanager" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199404 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="alertmanager" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199416 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy-metric" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199425 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy-metric" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199433 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="prom-label-proxy" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199441 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="prom-label-proxy" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199453 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="config-reloader" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199460 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="config-reloader" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199473 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="init-config-reloader" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199480 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="init-config-reloader" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199536 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy-web" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199547 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="alertmanager" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199555 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy-metric" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199566 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="config-reloader" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199574 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="kube-rbac-proxy" Jul 31 14:18:30.200504 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.199582 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" containerName="prom-label-proxy" Jul 31 14:18:30.207496 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.207377 2558 scope.go:117] "RemoveContainer" containerID="b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac" Jul 31 14:18:30.226257 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.226226 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:18:30.226391 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.226356 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.229533 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.229504 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 31 14:18:30.229670 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.229532 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 31 14:18:30.229670 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.229512 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 31 14:18:30.229670 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.229662 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 31 14:18:30.229850 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.229702 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 31 14:18:30.229850 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.229739 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 31 14:18:30.230029 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.229958 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 31 14:18:30.230160 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.230142 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 31 14:18:30.230323 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.230278 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rqwcl\"" Jul 31 14:18:30.242954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.242186 2558 scope.go:117] "RemoveContainer" containerID="cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc" Jul 31 14:18:30.245937 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.245658 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 31 14:18:30.251855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.251823 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.251981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.251867 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.251981 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.251899 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-config-volume\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.251974 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97b506e6-b571-409f-a6f2-87e791e96f59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252051 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252082 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b506e6-b571-409f-a6f2-87e791e96f59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252124 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-web-config\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252147 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpjm\" (UniqueName: \"kubernetes.io/projected/97b506e6-b571-409f-a6f2-87e791e96f59-kube-api-access-bzpjm\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252175 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97b506e6-b571-409f-a6f2-87e791e96f59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252205 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97b506e6-b571-409f-a6f2-87e791e96f59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252230 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252474 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252287 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97b506e6-b571-409f-a6f2-87e791e96f59-config-out\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.252474 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.252311 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.255884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.255861 2558 scope.go:117] "RemoveContainer" containerID="03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378" Jul 31 14:18:30.266038 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.266017 2558 scope.go:117] "RemoveContainer" containerID="9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6" Jul 31 14:18:30.266353 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:18:30.266334 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6\": container with ID starting with 9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6 not found: ID does not exist" containerID="9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6" Jul 31 14:18:30.266429 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.266362 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6"} err="failed to get container status \"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6\": rpc error: code = NotFound desc = could not find container \"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6\": container with ID starting with 9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6 not found: ID does not exist" Jul 31 14:18:30.266429 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.266380 2558 scope.go:117] "RemoveContainer" containerID="1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228" Jul 31 14:18:30.266727 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:18:30.266698 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228\": container with ID starting with 1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228 not found: ID does not exist" containerID="1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228" Jul 31 14:18:30.266967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.266733 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228"} err="failed to get container status \"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228\": rpc error: code = NotFound desc = could not find container \"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228\": container with ID starting with 1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228 not found: ID does not exist" Jul 31 14:18:30.266967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.266757 2558 scope.go:117] "RemoveContainer" containerID="5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99" Jul 31 14:18:30.267221 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:18:30.267091 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99\": container with ID starting with 5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99 not found: ID does not exist" containerID="5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99" Jul 31 14:18:30.267221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.267120 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99"} err="failed to get container status \"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99\": rpc error: code = NotFound desc = could not find container \"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99\": container with ID starting with 5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99 not found: ID does not exist" Jul 31 14:18:30.267221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.267139 2558 scope.go:117] "RemoveContainer" containerID="fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2" Jul 31 14:18:30.267515 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:18:30.267487 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2\": container with ID starting with fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2 not found: ID does not exist" containerID="fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2" Jul 31 14:18:30.267584 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.267521 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2"} err="failed to get container status \"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2\": rpc error: code = NotFound desc = could not find container \"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2\": container with ID starting with fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2 not found: ID does not exist" Jul 31 14:18:30.267584 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.267534 2558 scope.go:117] "RemoveContainer" containerID="b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac" Jul 31 14:18:30.267803 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:18:30.267785 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac\": container with ID starting with b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac not found: ID does not exist" containerID="b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac" Jul 31 14:18:30.267858 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.267807 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac"} err="failed to get container status \"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac\": rpc error: code = NotFound desc = could not find container \"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac\": container with ID starting with b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac not found: ID does not exist" Jul 31 14:18:30.267858 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.267821 2558 scope.go:117] "RemoveContainer" containerID="cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc" Jul 31 14:18:30.268095 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:18:30.268072 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc\": container with ID starting with cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc not found: ID does not exist" containerID="cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc" Jul 31 14:18:30.268203 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.268096 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc"} err="failed to get container status \"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc\": rpc error: code = NotFound desc = could not find container \"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc\": container with ID starting with cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc not found: ID does not exist" Jul 31 14:18:30.268203 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.268110 2558 scope.go:117] "RemoveContainer" containerID="03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378" Jul 31 14:18:30.268391 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:18:30.268371 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378\": container with ID starting with 03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378 not found: ID does not exist" containerID="03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378" Jul 31 14:18:30.268432 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.268397 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378"} err="failed to get container status \"03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378\": rpc error: code = NotFound desc = could not find container \"03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378\": container with ID starting with 03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378 not found: ID does not exist" Jul 31 14:18:30.268432 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.268412 2558 scope.go:117] "RemoveContainer" containerID="9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6" Jul 31 14:18:30.268668 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.268648 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6"} err="failed to get container status \"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6\": rpc error: code = NotFound desc = could not find container \"9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6\": container with ID starting with 9d921b49e689386d1c2302a8b927eaa4d0160c7f16dfcab1dfac5efd050f24b6 not found: ID does not exist" Jul 31 14:18:30.268668 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.268668 2558 scope.go:117] "RemoveContainer" containerID="1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228" Jul 31 14:18:30.268923 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.268890 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228"} err="failed to get container status \"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228\": rpc error: code = NotFound desc = could not find container \"1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228\": container with ID starting with 1fea5f3fba98aa0da7c8d88dfe8d2dea25bdd8596ce788907b5f57ca4169d228 not found: ID does not exist" Jul 31 14:18:30.268964 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.268925 2558 scope.go:117] "RemoveContainer" containerID="5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99" Jul 31 14:18:30.269200 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.269162 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99"} err="failed to get container status \"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99\": rpc error: code = NotFound desc = could not find container \"5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99\": container with ID starting with 5d89657d2f8ac76783bc64334ce7e8521cf83f3e76901a3496d41e815a18af99 not found: ID does not exist" Jul 31 14:18:30.269261 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.269203 2558 scope.go:117] "RemoveContainer" containerID="fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2" Jul 31 14:18:30.269460 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.269440 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2"} err="failed to get container status \"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2\": rpc error: code = NotFound desc = could not find container \"fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2\": container with ID starting with fe74f5071e06981144ce50f7ac610e632f4a726e9681321947157e30973479a2 not found: ID does not exist" Jul 31 14:18:30.269515 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.269461 2558 scope.go:117] "RemoveContainer" containerID="b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac" Jul 31 14:18:30.269694 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.269678 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac"} err="failed to get container status \"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac\": rpc error: code = NotFound desc = could not find container \"b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac\": container with ID starting with b09e19f38ff251079e7643f1c6ca5f8744376df7e162d6f82340b22976bee3ac not found: ID does not exist" Jul 31 14:18:30.269694 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.269694 2558 scope.go:117] "RemoveContainer" containerID="cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc" Jul 31 14:18:30.269891 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.269875 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc"} err="failed to get container status \"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc\": rpc error: code = NotFound desc = could not find container \"cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc\": container with ID starting with cbb3090699cf3bf0eb70050d7370fa3a8974dc6fbb9600ade5f33a83381fd7cc not found: ID does not exist" Jul 31 14:18:30.269951 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.269891 2558 scope.go:117] "RemoveContainer" containerID="03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378" Jul 31 14:18:30.270108 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.270092 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378"} err="failed to get container status \"03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378\": rpc error: code = NotFound desc = could not find container \"03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378\": container with ID starting with 03c964b99e014377f48f6f59088a064846f71b42ba124ed80bb1ab9364887378 not found: ID does not exist" Jul 31 14:18:30.353332 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353284 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353534 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b506e6-b571-409f-a6f2-87e791e96f59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353534 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353380 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-web-config\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353534 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353398 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpjm\" (UniqueName: \"kubernetes.io/projected/97b506e6-b571-409f-a6f2-87e791e96f59-kube-api-access-bzpjm\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353534 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353430 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97b506e6-b571-409f-a6f2-87e791e96f59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353534 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353457 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97b506e6-b571-409f-a6f2-87e791e96f59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353534 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353480 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353534 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97b506e6-b571-409f-a6f2-87e791e96f59-config-out\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353898 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353557 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353898 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353581 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353898 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353608 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353898 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353644 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-config-volume\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.353898 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.353681 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97b506e6-b571-409f-a6f2-87e791e96f59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.354733 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.354698 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b506e6-b571-409f-a6f2-87e791e96f59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.355051 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.355011 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97b506e6-b571-409f-a6f2-87e791e96f59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.355563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.355529 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97b506e6-b571-409f-a6f2-87e791e96f59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.357420 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.357066 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-web-config\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.357420 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.357368 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-config-volume\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.358322 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.357921 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97b506e6-b571-409f-a6f2-87e791e96f59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.358322 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.358106 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.358322 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.358213 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.358322 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.358292 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.358538 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.358521 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97b506e6-b571-409f-a6f2-87e791e96f59-config-out\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.358579 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.358556 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.359032 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.359012 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97b506e6-b571-409f-a6f2-87e791e96f59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.361494 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.361472 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpjm\" (UniqueName: \"kubernetes.io/projected/97b506e6-b571-409f-a6f2-87e791e96f59-kube-api-access-bzpjm\") pod \"alertmanager-main-0\" (UID: \"97b506e6-b571-409f-a6f2-87e791e96f59\") " pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.547979 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.547941 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 31 14:18:30.686689 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.686663 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 31 14:18:30.688825 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:18:30.688803 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b506e6_b571_409f_a6f2_87e791e96f59.slice/crio-a978c0b2bd60f572bbc36b7f21b83effd3c086b75d631b8f4c17a5dfb2550fa4 WatchSource:0}: Error finding container a978c0b2bd60f572bbc36b7f21b83effd3c086b75d631b8f4c17a5dfb2550fa4: Status 404 returned error can't find the container with id a978c0b2bd60f572bbc36b7f21b83effd3c086b75d631b8f4c17a5dfb2550fa4 Jul 31 14:18:30.982949 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:30.982852 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f25afae-727e-4702-9fec-e7d839578cc1" path="/var/lib/kubelet/pods/5f25afae-727e-4702-9fec-e7d839578cc1/volumes" Jul 31 14:18:31.151605 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:31.151568 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdtjx" event={"ID":"26920aff-943f-4177-8777-d186d5e99fc3","Type":"ContainerStarted","Data":"7208a78bc8cc3610b2450ed444f2590689db2de3915952f7a36af13ac2f563b4"} Jul 31 14:18:31.151809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:31.151641 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdtjx" event={"ID":"26920aff-943f-4177-8777-d186d5e99fc3","Type":"ContainerStarted","Data":"c4e2d9f0cfcc22cfc263bd612c0d6fd7a146b9fd25e971cceff13edd30c6b385"} Jul 31 14:18:31.154079 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:31.154053 2558 generic.go:356] "Generic (PLEG): container finished" podID="97b506e6-b571-409f-a6f2-87e791e96f59" containerID="b8e2a20665dca56bbcd739c784ba4c229a0c31480f4652e7d8df250a3bac50f7" exitCode=0 Jul 31 14:18:31.154175 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:31.154085 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97b506e6-b571-409f-a6f2-87e791e96f59","Type":"ContainerDied","Data":"b8e2a20665dca56bbcd739c784ba4c229a0c31480f4652e7d8df250a3bac50f7"} Jul 31 14:18:31.154175 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:31.154130 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97b506e6-b571-409f-a6f2-87e791e96f59","Type":"ContainerStarted","Data":"a978c0b2bd60f572bbc36b7f21b83effd3c086b75d631b8f4c17a5dfb2550fa4"} Jul 31 14:18:31.167770 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:31.167650 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kdtjx" podStartSLOduration=375.106706753 podStartE2EDuration="6m16.167632885s" totalImagesPullingTime="1.060926132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:18:31.166605068 +0000 UTC m=+376.674984133" watchObservedRunningTime="2026-07-31 14:18:31.167632885 +0000 UTC m=+376.676011952" Jul 31 14:18:31.249439 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:31.249407 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:31.351018 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:31.350988 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:32.160828 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.160790 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97b506e6-b571-409f-a6f2-87e791e96f59","Type":"ContainerStarted","Data":"dbef22a3d3a657938953b210298bd10816606f2da9b82e0d07f1471512e54b79"} Jul 31 14:18:32.161223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.160892 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97b506e6-b571-409f-a6f2-87e791e96f59","Type":"ContainerStarted","Data":"43e5e6256972edf5f7c9e706c8d4c50f4ef3e3f1ca02a154f6a8c23f627c7dde"} Jul 31 14:18:32.161223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.160940 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97b506e6-b571-409f-a6f2-87e791e96f59","Type":"ContainerStarted","Data":"68296a076c1d298b05a7775d1fcaf94b70a98d475ef132ea80978e5293420184"} Jul 31 14:18:32.161223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.160966 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97b506e6-b571-409f-a6f2-87e791e96f59","Type":"ContainerStarted","Data":"44bf7484c6ba24120100f36a96573f8afa6ce889d4e8d8905050054c8d9f2b61"} Jul 31 14:18:32.161223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.160989 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97b506e6-b571-409f-a6f2-87e791e96f59","Type":"ContainerStarted","Data":"f308fe532289f0e1d298e26440bc5c56828a113aaad6726ce5d9940df30964cc"} Jul 31 14:18:32.161223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.161013 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97b506e6-b571-409f-a6f2-87e791e96f59","Type":"ContainerStarted","Data":"8f9ed090d0faa9893e536efc8901b9355d384979726d15b5780669debb005068"} Jul 31 14:18:32.188401 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.188345 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.17386524 podStartE2EDuration="2.188329013s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.463773ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:18:32.185891002 +0000 UTC m=+377.694270067" watchObservedRunningTime="2026-07-31 14:18:32.188329013 +0000 UTC m=+377.696708078" Jul 31 14:18:32.316231 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.316189 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-bd9997cfc-j4jvh"] Jul 31 14:18:32.319584 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.319565 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.323278 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.323251 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 31 14:18:32.323421 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.323252 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 31 14:18:32.323421 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.323407 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c8dq9\"" Jul 31 14:18:32.323542 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.323486 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 31 14:18:32.323778 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.323746 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 31 14:18:32.323778 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.323764 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 31 14:18:32.327767 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.327664 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 31 14:18:32.333038 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.333015 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bd9997cfc-j4jvh"] Jul 31 14:18:32.369336 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.369298 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-federate-client-tls\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.369336 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.369337 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-metrics-client-ca\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.369558 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.369361 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-secret-telemeter-client\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.369558 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.369447 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-serving-certs-ca-bundle\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.369558 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.369484 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.369558 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.369512 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-telemeter-client-tls\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.369558 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.369544 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.369725 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.369568 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhsw\" (UniqueName: \"kubernetes.io/projected/8af786f7-62f4-4021-b74d-3aa1a00268c1-kube-api-access-4mhsw\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.470502 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.470397 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-federate-client-tls\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.470502 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.470459 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-metrics-client-ca\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.470735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.470513 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-secret-telemeter-client\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.470735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.470556 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-serving-certs-ca-bundle\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.470735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.470591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.470735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.470626 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-telemeter-client-tls\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.470735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.470663 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.470735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.470695 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhsw\" (UniqueName: \"kubernetes.io/projected/8af786f7-62f4-4021-b74d-3aa1a00268c1-kube-api-access-4mhsw\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.471973 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.471941 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-serving-certs-ca-bundle\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.472108 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.472087 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.472425 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.472377 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8af786f7-62f4-4021-b74d-3aa1a00268c1-metrics-client-ca\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.474161 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.474138 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.474370 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.474350 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-secret-telemeter-client\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.474406 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.474355 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-federate-client-tls\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.474406 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.474382 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8af786f7-62f4-4021-b74d-3aa1a00268c1-telemeter-client-tls\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.478695 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.478666 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhsw\" (UniqueName: \"kubernetes.io/projected/8af786f7-62f4-4021-b74d-3aa1a00268c1-kube-api-access-4mhsw\") pod \"telemeter-client-bd9997cfc-j4jvh\" (UID: \"8af786f7-62f4-4021-b74d-3aa1a00268c1\") " pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.631306 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.631015 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" Jul 31 14:18:32.784223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:32.784189 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bd9997cfc-j4jvh"] Jul 31 14:18:32.788385 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:18:32.788354 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af786f7_62f4_4021_b74d_3aa1a00268c1.slice/crio-b4837be75aabcb7728f66b5eddf2ec73a0e7c1bd871d4315f180ae036ac7a006 WatchSource:0}: Error finding container b4837be75aabcb7728f66b5eddf2ec73a0e7c1bd871d4315f180ae036ac7a006: Status 404 returned error can't find the container with id b4837be75aabcb7728f66b5eddf2ec73a0e7c1bd871d4315f180ae036ac7a006 Jul 31 14:18:33.166583 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:33.166547 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" event={"ID":"8af786f7-62f4-4021-b74d-3aa1a00268c1","Type":"ContainerStarted","Data":"b4837be75aabcb7728f66b5eddf2ec73a0e7c1bd871d4315f180ae036ac7a006"} Jul 31 14:18:35.176237 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:35.176207 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" event={"ID":"8af786f7-62f4-4021-b74d-3aa1a00268c1","Type":"ContainerStarted","Data":"4c3dda980122d479d3e2c67bc987f2fdbe1801fd8c1ca42ce3deed2272139818"} Jul 31 14:18:36.182137 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.182094 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" event={"ID":"8af786f7-62f4-4021-b74d-3aa1a00268c1","Type":"ContainerStarted","Data":"7ad7144c528355c17aed1c2735bb216cf0a7b672f8fe4a8b55512f9bef65110d"} Jul 31 14:18:36.182511 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.182166 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" event={"ID":"8af786f7-62f4-4021-b74d-3aa1a00268c1","Type":"ContainerStarted","Data":"9e9a05fbd3ed4858aa084eba67bcab1bcb93a19c314c304c8abc29ccd8e33d2f"} Jul 31 14:18:36.205313 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.205262 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-bd9997cfc-j4jvh" podStartSLOduration=1.9332865510000001 podStartE2EDuration="4.205240619s" totalImagesPullingTime="2.271954068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:18:36.203487773 +0000 UTC m=+381.711866839" watchObservedRunningTime="2026-07-31 14:18:36.205240619 +0000 UTC m=+381.713619684" Jul 31 14:18:36.282118 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.282084 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77c56665dc-szzwp"] Jul 31 14:18:36.285455 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.285435 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.295227 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.295203 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c56665dc-szzwp"] Jul 31 14:18:36.301064 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.301040 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-service-ca\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.301179 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.301090 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-oauth-serving-cert\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.301179 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.301123 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qll4j\" (UniqueName: \"kubernetes.io/projected/157bd253-2b93-4d41-8674-14fa9627ec5b-kube-api-access-qll4j\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.301315 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.301178 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-serving-cert\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.301315 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.301210 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-oauth-config\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.301315 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.301269 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-console-config\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.301315 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.301310 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-trusted-ca-bundle\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.339544 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.339507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:36.389649 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.389612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:36.401714 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.401676 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qll4j\" (UniqueName: \"kubernetes.io/projected/157bd253-2b93-4d41-8674-14fa9627ec5b-kube-api-access-qll4j\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.401885 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.401724 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-serving-cert\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.401885 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.401747 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-oauth-config\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.401885 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.401779 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-console-config\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.401885 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.401819 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-trusted-ca-bundle\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.401885 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.401847 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-service-ca\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.402162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.401893 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-oauth-serving-cert\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.402743 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.402712 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-oauth-serving-cert\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.402872 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.402769 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-service-ca\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.402970 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.402864 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-console-config\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.403038 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.402991 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-trusted-ca-bundle\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.405176 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.404777 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-serving-cert\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.405592 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.405571 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-oauth-config\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.409996 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.409974 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qll4j\" (UniqueName: \"kubernetes.io/projected/157bd253-2b93-4d41-8674-14fa9627ec5b-kube-api-access-qll4j\") pod \"console-77c56665dc-szzwp\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.478321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.478239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:36.574204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.574165 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:36.594669 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.594626 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:36.747479 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:36.747450 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c56665dc-szzwp"] Jul 31 14:18:37.085921 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:37.085822 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:37.187134 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:37.187095 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c56665dc-szzwp" event={"ID":"157bd253-2b93-4d41-8674-14fa9627ec5b","Type":"ContainerStarted","Data":"8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b"} Jul 31 14:18:37.187576 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:37.187150 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c56665dc-szzwp" event={"ID":"157bd253-2b93-4d41-8674-14fa9627ec5b","Type":"ContainerStarted","Data":"631c6729276aa41de40ff2f868cecbef75b0861804d38b8172a760de05b51a53"} Jul 31 14:18:37.207013 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:37.206965 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77c56665dc-szzwp" podStartSLOduration=1.206950842 podStartE2EDuration="1.206950842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:18:37.205471699 +0000 UTC m=+382.713850765" watchObservedRunningTime="2026-07-31 14:18:37.206950842 +0000 UTC m=+382.715329908" Jul 31 14:18:44.935509 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:44.935467 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:44.989072 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:44.989031 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:46.595682 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:46.595648 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:46.596178 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:46.595699 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:46.600326 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:46.600303 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:47.240005 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:47.239975 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:18:47.281321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:47.281291 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579565665b-wbl52"] Jul 31 14:18:47.469425 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:47.469387 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:18:47.519216 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:18:47.519131 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:19:12.302239 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.302176 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-579565665b-wbl52" podUID="ee03d7a9-eeac-4784-ab80-2504a1177760" containerName="console" containerID="cri-o://25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef" gracePeriod=15 Jul 31 14:19:12.548837 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.548803 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579565665b-wbl52" Jul 31 14:19:12.710747 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.710695 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmhpb\" (UniqueName: \"kubernetes.io/projected/ee03d7a9-eeac-4784-ab80-2504a1177760-kube-api-access-zmhpb\") pod \"ee03d7a9-eeac-4784-ab80-2504a1177760\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " Jul 31 14:19:12.710747 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.710738 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-console-config\") pod \"ee03d7a9-eeac-4784-ab80-2504a1177760\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " Jul 31 14:19:12.710747 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.710756 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-serving-cert\") pod \"ee03d7a9-eeac-4784-ab80-2504a1177760\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " Jul 31 14:19:12.711095 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.710785 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-oauth-serving-cert\") pod \"ee03d7a9-eeac-4784-ab80-2504a1177760\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " Jul 31 14:19:12.711095 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.710968 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-oauth-config\") pod \"ee03d7a9-eeac-4784-ab80-2504a1177760\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " Jul 31 14:19:12.711095 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.711039 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-service-ca\") pod \"ee03d7a9-eeac-4784-ab80-2504a1177760\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " Jul 31 14:19:12.711234 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.711104 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-trusted-ca-bundle\") pod \"ee03d7a9-eeac-4784-ab80-2504a1177760\" (UID: \"ee03d7a9-eeac-4784-ab80-2504a1177760\") " Jul 31 14:19:12.711234 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.711190 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-console-config" (OuterVolumeSpecName: "console-config") pod "ee03d7a9-eeac-4784-ab80-2504a1177760" (UID: "ee03d7a9-eeac-4784-ab80-2504a1177760"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:19:12.711234 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.711221 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee03d7a9-eeac-4784-ab80-2504a1177760" (UID: "ee03d7a9-eeac-4784-ab80-2504a1177760"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:19:12.711563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.711538 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ee03d7a9-eeac-4784-ab80-2504a1177760" (UID: "ee03d7a9-eeac-4784-ab80-2504a1177760"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:19:12.711563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.711552 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-console-config\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:19:12.711707 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.711574 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-oauth-serving-cert\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:19:12.711707 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.711551 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee03d7a9-eeac-4784-ab80-2504a1177760" (UID: "ee03d7a9-eeac-4784-ab80-2504a1177760"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:19:12.713188 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.713158 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee03d7a9-eeac-4784-ab80-2504a1177760" (UID: "ee03d7a9-eeac-4784-ab80-2504a1177760"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:19:12.713311 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.713250 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee03d7a9-eeac-4784-ab80-2504a1177760" (UID: "ee03d7a9-eeac-4784-ab80-2504a1177760"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:19:12.713369 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.713331 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee03d7a9-eeac-4784-ab80-2504a1177760-kube-api-access-zmhpb" (OuterVolumeSpecName: "kube-api-access-zmhpb") pod "ee03d7a9-eeac-4784-ab80-2504a1177760" (UID: "ee03d7a9-eeac-4784-ab80-2504a1177760"). InnerVolumeSpecName "kube-api-access-zmhpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:19:12.812861 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.812823 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhpb\" (UniqueName: \"kubernetes.io/projected/ee03d7a9-eeac-4784-ab80-2504a1177760-kube-api-access-zmhpb\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:19:12.812861 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.812853 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-serving-cert\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:19:12.812861 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.812865 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee03d7a9-eeac-4784-ab80-2504a1177760-console-oauth-config\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:19:12.813118 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.812880 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:19:12.813118 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:12.812892 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee03d7a9-eeac-4784-ab80-2504a1177760-trusted-ca-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:19:13.340961 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.340858 2558 generic.go:356] "Generic (PLEG): container finished" podID="ee03d7a9-eeac-4784-ab80-2504a1177760" containerID="25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef" exitCode=0 Jul 31 14:19:13.340961 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.340916 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579565665b-wbl52" Jul 31 14:19:13.340961 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.340944 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579565665b-wbl52" event={"ID":"ee03d7a9-eeac-4784-ab80-2504a1177760","Type":"ContainerDied","Data":"25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef"} Jul 31 14:19:13.341491 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.340989 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579565665b-wbl52" event={"ID":"ee03d7a9-eeac-4784-ab80-2504a1177760","Type":"ContainerDied","Data":"fd545b40522fcc944536f1d1e89ce79228f1ed5417c32d339ecc07ff18ac0538"} Jul 31 14:19:13.341491 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.341010 2558 scope.go:117] "RemoveContainer" containerID="25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef" Jul 31 14:19:13.349202 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.349179 2558 scope.go:117] "RemoveContainer" containerID="25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef" Jul 31 14:19:13.349487 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:19:13.349468 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef\": container with ID starting with 25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef not found: ID does not exist" containerID="25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef" Jul 31 14:19:13.349561 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.349500 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef"} err="failed to get container status \"25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef\": rpc error: code = NotFound desc = could not find container \"25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef\": container with ID starting with 25ab2ef8f976a5ef38b47c16348827a96af3cb5acf6865310a1872242a7151ef not found: ID does not exist" Jul 31 14:19:13.358243 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.358221 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579565665b-wbl52"] Jul 31 14:19:13.361732 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:13.361713 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-579565665b-wbl52"] Jul 31 14:19:14.982211 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:14.982177 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee03d7a9-eeac-4784-ab80-2504a1177760" path="/var/lib/kubelet/pods/ee03d7a9-eeac-4784-ab80-2504a1177760/volumes" Jul 31 14:19:36.464628 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:36.464590 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:19:36.514691 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:36.514654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:19:42.466256 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:42.466210 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:19:42.513950 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:42.513879 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:19:54.288035 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:54.287998 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:19:54.341663 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:54.341632 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:19:56.543129 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:56.543091 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:19:56.594295 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:19:56.594253 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:20:03.238555 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:03.238517 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:20:03.338813 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:03.338775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:20:04.133269 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:04.133232 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:20:07.208010 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:07.207975 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77c56665dc-szzwp"] Jul 31 14:20:07.298445 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:07.298412 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:20:07.349140 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:07.349109 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:20:07.404948 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:07.404888 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:20:11.684745 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:11.684710 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:20:32.232250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.232192 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77c56665dc-szzwp" podUID="157bd253-2b93-4d41-8674-14fa9627ec5b" containerName="console" containerID="cri-o://8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b" gracePeriod=15 Jul 31 14:20:32.467228 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.467203 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:20:32.534316 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534242 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qll4j\" (UniqueName: \"kubernetes.io/projected/157bd253-2b93-4d41-8674-14fa9627ec5b-kube-api-access-qll4j\") pod \"157bd253-2b93-4d41-8674-14fa9627ec5b\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " Jul 31 14:20:32.534316 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534285 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-service-ca\") pod \"157bd253-2b93-4d41-8674-14fa9627ec5b\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " Jul 31 14:20:32.534512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534319 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-console-config\") pod \"157bd253-2b93-4d41-8674-14fa9627ec5b\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " Jul 31 14:20:32.534512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534348 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-oauth-serving-cert\") pod \"157bd253-2b93-4d41-8674-14fa9627ec5b\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " Jul 31 14:20:32.534512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534368 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-trusted-ca-bundle\") pod \"157bd253-2b93-4d41-8674-14fa9627ec5b\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " Jul 31 14:20:32.534512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534411 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-serving-cert\") pod \"157bd253-2b93-4d41-8674-14fa9627ec5b\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " Jul 31 14:20:32.534512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534436 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-oauth-config\") pod \"157bd253-2b93-4d41-8674-14fa9627ec5b\" (UID: \"157bd253-2b93-4d41-8674-14fa9627ec5b\") " Jul 31 14:20:32.534818 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534772 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-service-ca" (OuterVolumeSpecName: "service-ca") pod "157bd253-2b93-4d41-8674-14fa9627ec5b" (UID: "157bd253-2b93-4d41-8674-14fa9627ec5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:20:32.534964 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534921 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-console-config" (OuterVolumeSpecName: "console-config") pod "157bd253-2b93-4d41-8674-14fa9627ec5b" (UID: "157bd253-2b93-4d41-8674-14fa9627ec5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:20:32.535019 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534978 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "157bd253-2b93-4d41-8674-14fa9627ec5b" (UID: "157bd253-2b93-4d41-8674-14fa9627ec5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:20:32.535019 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.534990 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "157bd253-2b93-4d41-8674-14fa9627ec5b" (UID: "157bd253-2b93-4d41-8674-14fa9627ec5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:20:32.536512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.536484 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "157bd253-2b93-4d41-8674-14fa9627ec5b" (UID: "157bd253-2b93-4d41-8674-14fa9627ec5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:20:32.536600 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.536558 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157bd253-2b93-4d41-8674-14fa9627ec5b-kube-api-access-qll4j" (OuterVolumeSpecName: "kube-api-access-qll4j") pod "157bd253-2b93-4d41-8674-14fa9627ec5b" (UID: "157bd253-2b93-4d41-8674-14fa9627ec5b"). InnerVolumeSpecName "kube-api-access-qll4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:20:32.536648 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.536589 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "157bd253-2b93-4d41-8674-14fa9627ec5b" (UID: "157bd253-2b93-4d41-8674-14fa9627ec5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:20:32.635189 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.635161 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-console-config\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:20:32.635189 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.635188 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-oauth-serving-cert\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:20:32.635414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.635204 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-trusted-ca-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:20:32.635414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.635219 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-serving-cert\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:20:32.635414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.635228 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/157bd253-2b93-4d41-8674-14fa9627ec5b-console-oauth-config\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:20:32.635414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.635236 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qll4j\" (UniqueName: \"kubernetes.io/projected/157bd253-2b93-4d41-8674-14fa9627ec5b-kube-api-access-qll4j\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:20:32.635414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.635246 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/157bd253-2b93-4d41-8674-14fa9627ec5b-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:20:32.644666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.644632 2558 generic.go:356] "Generic (PLEG): container finished" podID="157bd253-2b93-4d41-8674-14fa9627ec5b" containerID="8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b" exitCode=0 Jul 31 14:20:32.644808 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.644692 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c56665dc-szzwp" Jul 31 14:20:32.644808 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.644725 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c56665dc-szzwp" event={"ID":"157bd253-2b93-4d41-8674-14fa9627ec5b","Type":"ContainerDied","Data":"8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b"} Jul 31 14:20:32.644876 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.644812 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c56665dc-szzwp" event={"ID":"157bd253-2b93-4d41-8674-14fa9627ec5b","Type":"ContainerDied","Data":"631c6729276aa41de40ff2f868cecbef75b0861804d38b8172a760de05b51a53"} Jul 31 14:20:32.644876 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.644839 2558 scope.go:117] "RemoveContainer" containerID="8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b" Jul 31 14:20:32.653511 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.653490 2558 scope.go:117] "RemoveContainer" containerID="8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b" Jul 31 14:20:32.653769 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:20:32.653751 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b\": container with ID starting with 8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b not found: ID does not exist" containerID="8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b" Jul 31 14:20:32.653820 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.653777 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b"} err="failed to get container status \"8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b\": rpc error: code = NotFound desc = could not find container \"8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b\": container with ID starting with 8ba2e196afa48fab22c9a711b8a5e8d89ea68a75a7fb5ec835acc83ba2e0a04b not found: ID does not exist" Jul 31 14:20:32.664637 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.664615 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77c56665dc-szzwp"] Jul 31 14:20:32.667707 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.667685 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77c56665dc-szzwp"] Jul 31 14:20:32.982527 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:20:32.982491 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="157bd253-2b93-4d41-8674-14fa9627ec5b" path="/var/lib/kubelet/pods/157bd253-2b93-4d41-8674-14fa9627ec5b/volumes" Jul 31 14:21:11.904078 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:11.903986 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:21:12.002695 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.002660 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:21:12.524361 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.524324 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr"] Jul 31 14:21:12.524617 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.524605 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee03d7a9-eeac-4784-ab80-2504a1177760" containerName="console" Jul 31 14:21:12.524617 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.524618 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee03d7a9-eeac-4784-ab80-2504a1177760" containerName="console" Jul 31 14:21:12.524732 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.524640 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157bd253-2b93-4d41-8674-14fa9627ec5b" containerName="console" Jul 31 14:21:12.524732 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.524646 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="157bd253-2b93-4d41-8674-14fa9627ec5b" containerName="console" Jul 31 14:21:12.524732 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.524693 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee03d7a9-eeac-4784-ab80-2504a1177760" containerName="console" Jul 31 14:21:12.524732 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.524701 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="157bd253-2b93-4d41-8674-14fa9627ec5b" containerName="console" Jul 31 14:21:12.527737 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.527718 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.530645 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.530620 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fdsgp\"" Jul 31 14:21:12.530768 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.530646 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 14:21:12.531863 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.531846 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 14:21:12.536391 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.536366 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr"] Jul 31 14:21:12.649133 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.649095 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.649320 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.649150 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2xb\" (UniqueName: \"kubernetes.io/projected/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-kube-api-access-lq2xb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.649320 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.649272 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.750068 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.750025 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.750252 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.750114 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2xb\" (UniqueName: \"kubernetes.io/projected/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-kube-api-access-lq2xb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.750252 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.750187 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.750426 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.750402 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.750509 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.750489 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.758656 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.758632 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2xb\" (UniqueName: \"kubernetes.io/projected/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-kube-api-access-lq2xb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.838376 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.838278 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:12.967810 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:12.967776 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr"] Jul 31 14:21:12.970853 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:21:12.970813 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fea61d_9f49_47a0_8b18_6fe7ff0d57f5.slice/crio-6e060ebb527640523334723fe33704ae82981f66a17c56f8266fbe61aeb35b74 WatchSource:0}: Error finding container 6e060ebb527640523334723fe33704ae82981f66a17c56f8266fbe61aeb35b74: Status 404 returned error can't find the container with id 6e060ebb527640523334723fe33704ae82981f66a17c56f8266fbe61aeb35b74 Jul 31 14:21:13.798703 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:13.798660 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" event={"ID":"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5","Type":"ContainerStarted","Data":"6e060ebb527640523334723fe33704ae82981f66a17c56f8266fbe61aeb35b74"} Jul 31 14:21:19.830653 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:19.830558 2558 generic.go:356] "Generic (PLEG): container finished" podID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerID="70806e860293f585c6bfd68fc8445b73029d4c7f638403315ab190c1d3819185" exitCode=0 Jul 31 14:21:19.830653 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:19.830646 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" event={"ID":"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5","Type":"ContainerDied","Data":"70806e860293f585c6bfd68fc8445b73029d4c7f638403315ab190c1d3819185"} Jul 31 14:21:30.274318 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:21:30.274211 2558 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/cert-manager/cert-manager-operator-bundle@sha256=00161fb5d18db09b6d77bbddb1babe54dc1a3cf12a7eced147352ad3608754ce/signature-1: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="registry.redhat.io/cert-manager/cert-manager-operator-bundle@sha256:00161fb5d18db09b6d77bbddb1babe54dc1a3cf12a7eced147352ad3608754ce" Jul 31 14:21:30.274794 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:21:30.274348 2558 kuberuntime_manager.go:1459] "Unhandled Error" err="init container pull start failed in pod 685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr_openshift-marketplace(b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5): ErrImagePull: unable to pull image or OCI artifact: pull image err: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/cert-manager/cert-manager-operator-bundle@sha256=00161fb5d18db09b6d77bbddb1babe54dc1a3cf12a7eced147352ad3608754ce/signature-1: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 31 14:21:30.274794 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:21:30.274396 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"pull\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/cert-manager/cert-manager-operator-bundle@sha256=00161fb5d18db09b6d77bbddb1babe54dc1a3cf12a7eced147352ad3608754ce/signature-1: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" Jul 31 14:21:30.873857 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:21:30.873823 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"pull\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/cert-manager/cert-manager-operator-bundle@sha256:00161fb5d18db09b6d77bbddb1babe54dc1a3cf12a7eced147352ad3608754ce\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/cert-manager/cert-manager-operator-bundle@sha256=00161fb5d18db09b6d77bbddb1babe54dc1a3cf12a7eced147352ad3608754ce/signature-1: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" Jul 31 14:21:44.924331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:44.924300 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" event={"ID":"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5","Type":"ContainerStarted","Data":"be31388e7e31168d3b08b87051467e4e18f7c11e6c2083fd34a84da5d1c8d9ec"} Jul 31 14:21:45.929919 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:45.929867 2558 generic.go:356] "Generic (PLEG): container finished" podID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerID="be31388e7e31168d3b08b87051467e4e18f7c11e6c2083fd34a84da5d1c8d9ec" exitCode=0 Jul 31 14:21:45.930281 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:45.929945 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" event={"ID":"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5","Type":"ContainerDied","Data":"be31388e7e31168d3b08b87051467e4e18f7c11e6c2083fd34a84da5d1c8d9ec"} Jul 31 14:21:52.961180 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:52.961098 2558 generic.go:356] "Generic (PLEG): container finished" podID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerID="381595305e84eb7e5c8149e69d16cea5bca3584a437308a256f6afb82d6f6ca7" exitCode=0 Jul 31 14:21:52.961553 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:52.961171 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" event={"ID":"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5","Type":"ContainerDied","Data":"381595305e84eb7e5c8149e69d16cea5bca3584a437308a256f6afb82d6f6ca7"} Jul 31 14:21:54.085705 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.085682 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:54.177674 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.177630 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-util\") pod \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " Jul 31 14:21:54.177849 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.177690 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-bundle\") pod \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " Jul 31 14:21:54.177849 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.177715 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq2xb\" (UniqueName: \"kubernetes.io/projected/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-kube-api-access-lq2xb\") pod \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\" (UID: \"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5\") " Jul 31 14:21:54.178381 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.178346 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-bundle" (OuterVolumeSpecName: "bundle") pod "b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" (UID: "b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:21:54.179895 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.179865 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-kube-api-access-lq2xb" (OuterVolumeSpecName: "kube-api-access-lq2xb") pod "b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" (UID: "b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5"). InnerVolumeSpecName "kube-api-access-lq2xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:21:54.182514 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.182492 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-util" (OuterVolumeSpecName: "util") pod "b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" (UID: "b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:21:54.279311 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.279204 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:21:54.279311 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.279252 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq2xb\" (UniqueName: \"kubernetes.io/projected/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-kube-api-access-lq2xb\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:21:54.279311 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.279266 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5-util\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:21:54.969940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.969898 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" Jul 31 14:21:54.970129 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.969891 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bz7vr" event={"ID":"b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5","Type":"ContainerDied","Data":"6e060ebb527640523334723fe33704ae82981f66a17c56f8266fbe61aeb35b74"} Jul 31 14:21:54.970129 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:54.970052 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e060ebb527640523334723fe33704ae82981f66a17c56f8266fbe61aeb35b74" Jul 31 14:21:58.643471 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:21:58.643430 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:00.984462 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:00.984425 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:02.410994 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:02.410957 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:02.511215 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:02.511175 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:06.581266 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:06.581223 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:06.681542 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:06.681511 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:07.008879 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:07.008840 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:07.075948 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:07.075895 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:29.962057 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:29.962016 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:30.065481 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:30.065450 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:30.222405 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:30.222317 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:31.428684 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.428649 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p"] Jul 31 14:22:31.429074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.428935 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerName="extract" Jul 31 14:22:31.429074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.428946 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerName="extract" Jul 31 14:22:31.429074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.428956 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerName="util" Jul 31 14:22:31.429074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.428962 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerName="util" Jul 31 14:22:31.429074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.428979 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerName="pull" Jul 31 14:22:31.429074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.428984 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerName="pull" Jul 31 14:22:31.429074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.429042 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2fea61d-9f49-47a0-8b18-6fe7ff0d57f5" containerName="extract" Jul 31 14:22:31.430895 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.430878 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.433698 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.433673 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 14:22:31.435054 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.435035 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 14:22:31.435151 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.435037 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fdsgp\"" Jul 31 14:22:31.439729 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.439706 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p"] Jul 31 14:22:31.576259 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.576231 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.576401 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.576262 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkvk9\" (UniqueName: \"kubernetes.io/projected/08b8c271-5548-4b1b-9a12-b2f26e44f97b-kube-api-access-bkvk9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.576401 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.576295 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.677619 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.677572 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.677619 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.677626 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkvk9\" (UniqueName: \"kubernetes.io/projected/08b8c271-5548-4b1b-9a12-b2f26e44f97b-kube-api-access-bkvk9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.677832 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.677677 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.678105 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.678079 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.678105 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.678096 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.689198 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.689141 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkvk9\" (UniqueName: \"kubernetes.io/projected/08b8c271-5548-4b1b-9a12-b2f26e44f97b-kube-api-access-bkvk9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.739884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.739840 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:31.862007 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:31.861956 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p"] Jul 31 14:22:31.864538 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:22:31.864506 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b8c271_5548_4b1b_9a12_b2f26e44f97b.slice/crio-643e37d01a8433ce73c67a127bd4c689e58e3a62c39eec796d96cbed04ecfe86 WatchSource:0}: Error finding container 643e37d01a8433ce73c67a127bd4c689e58e3a62c39eec796d96cbed04ecfe86: Status 404 returned error can't find the container with id 643e37d01a8433ce73c67a127bd4c689e58e3a62c39eec796d96cbed04ecfe86 Jul 31 14:22:32.104864 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.104829 2558 generic.go:356] "Generic (PLEG): container finished" podID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerID="304fd22e098fceac8092f9a3d9b95f0941aea018c3135eb26ea2439a691b3482" exitCode=0 Jul 31 14:22:32.105164 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.104880 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" event={"ID":"08b8c271-5548-4b1b-9a12-b2f26e44f97b","Type":"ContainerDied","Data":"304fd22e098fceac8092f9a3d9b95f0941aea018c3135eb26ea2439a691b3482"} Jul 31 14:22:32.105164 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.104975 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" event={"ID":"08b8c271-5548-4b1b-9a12-b2f26e44f97b","Type":"ContainerStarted","Data":"643e37d01a8433ce73c67a127bd4c689e58e3a62c39eec796d96cbed04ecfe86"} Jul 31 14:22:32.214207 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.214117 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:32.305037 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.304997 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:32.718690 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.718657 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:32.923009 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.922974 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx"] Jul 31 14:22:32.925280 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.925263 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" Jul 31 14:22:32.928095 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.928076 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fp5bg\"" Jul 31 14:22:32.928289 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.928274 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 31 14:22:32.929735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.929721 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 31 14:22:32.934341 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:32.934320 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx"] Jul 31 14:22:33.089253 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:33.089168 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2z5\" (UniqueName: \"kubernetes.io/projected/08612026-960a-44c1-9638-0f0e87a9d148-kube-api-access-gn2z5\") pod \"cert-manager-cainjector-559cdd8c87-4fpdx\" (UID: \"08612026-960a-44c1-9638-0f0e87a9d148\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" Jul 31 14:22:33.089253 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:33.089211 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08612026-960a-44c1-9638-0f0e87a9d148-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4fpdx\" (UID: \"08612026-960a-44c1-9638-0f0e87a9d148\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" Jul 31 14:22:33.190373 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:33.190343 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2z5\" (UniqueName: \"kubernetes.io/projected/08612026-960a-44c1-9638-0f0e87a9d148-kube-api-access-gn2z5\") pod \"cert-manager-cainjector-559cdd8c87-4fpdx\" (UID: \"08612026-960a-44c1-9638-0f0e87a9d148\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" Jul 31 14:22:33.190525 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:33.190382 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08612026-960a-44c1-9638-0f0e87a9d148-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4fpdx\" (UID: \"08612026-960a-44c1-9638-0f0e87a9d148\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" Jul 31 14:22:33.199006 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:33.198977 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08612026-960a-44c1-9638-0f0e87a9d148-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4fpdx\" (UID: \"08612026-960a-44c1-9638-0f0e87a9d148\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" Jul 31 14:22:33.199115 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:33.199050 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2z5\" (UniqueName: \"kubernetes.io/projected/08612026-960a-44c1-9638-0f0e87a9d148-kube-api-access-gn2z5\") pod \"cert-manager-cainjector-559cdd8c87-4fpdx\" (UID: \"08612026-960a-44c1-9638-0f0e87a9d148\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" Jul 31 14:22:33.244711 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:33.244688 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" Jul 31 14:22:33.363319 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:33.363293 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx"] Jul 31 14:22:33.365186 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:22:33.365158 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08612026_960a_44c1_9638_0f0e87a9d148.slice/crio-a66536a9675cd345487ee479ad0da14b8ff1a4d6cc683d53a378fa82f8b7a7fd WatchSource:0}: Error finding container a66536a9675cd345487ee479ad0da14b8ff1a4d6cc683d53a378fa82f8b7a7fd: Status 404 returned error can't find the container with id a66536a9675cd345487ee479ad0da14b8ff1a4d6cc683d53a378fa82f8b7a7fd Jul 31 14:22:34.115289 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:34.115247 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" event={"ID":"08612026-960a-44c1-9638-0f0e87a9d148","Type":"ContainerStarted","Data":"a66536a9675cd345487ee479ad0da14b8ff1a4d6cc683d53a378fa82f8b7a7fd"} Jul 31 14:22:36.129491 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.129454 2558 generic.go:356] "Generic (PLEG): container finished" podID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerID="c43a57946df72c07095ef090ae11faa681116575a723c6f9bf97035d39fa3db4" exitCode=0 Jul 31 14:22:36.129998 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.129529 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" event={"ID":"08b8c271-5548-4b1b-9a12-b2f26e44f97b","Type":"ContainerDied","Data":"c43a57946df72c07095ef090ae11faa681116575a723c6f9bf97035d39fa3db4"} Jul 31 14:22:36.683789 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.681706 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hr9hd"] Jul 31 14:22:36.685206 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.685078 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:36.688290 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.688265 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-z6crl\"" Jul 31 14:22:36.691769 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.691743 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hr9hd"] Jul 31 14:22:36.819512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.819480 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532f707f-36b4-466f-8dce-e81bff34777e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hr9hd\" (UID: \"532f707f-36b4-466f-8dce-e81bff34777e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:36.819702 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.819557 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6d25\" (UniqueName: \"kubernetes.io/projected/532f707f-36b4-466f-8dce-e81bff34777e-kube-api-access-t6d25\") pod \"cert-manager-webhook-6486ffd997-hr9hd\" (UID: \"532f707f-36b4-466f-8dce-e81bff34777e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:36.921005 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.920968 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532f707f-36b4-466f-8dce-e81bff34777e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hr9hd\" (UID: \"532f707f-36b4-466f-8dce-e81bff34777e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:36.921174 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.921148 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6d25\" (UniqueName: \"kubernetes.io/projected/532f707f-36b4-466f-8dce-e81bff34777e-kube-api-access-t6d25\") pod \"cert-manager-webhook-6486ffd997-hr9hd\" (UID: \"532f707f-36b4-466f-8dce-e81bff34777e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:36.929351 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.929321 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/532f707f-36b4-466f-8dce-e81bff34777e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hr9hd\" (UID: \"532f707f-36b4-466f-8dce-e81bff34777e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:36.929351 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.929346 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6d25\" (UniqueName: \"kubernetes.io/projected/532f707f-36b4-466f-8dce-e81bff34777e-kube-api-access-t6d25\") pod \"cert-manager-webhook-6486ffd997-hr9hd\" (UID: \"532f707f-36b4-466f-8dce-e81bff34777e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:36.997996 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:36.997922 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:37.137630 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:37.137593 2558 generic.go:356] "Generic (PLEG): container finished" podID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerID="12227c7eed281f6b42957f062ad78b80d6d2b03728a4c3fe5a763ea19ee92f60" exitCode=0 Jul 31 14:22:37.138089 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:37.137655 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" event={"ID":"08b8c271-5548-4b1b-9a12-b2f26e44f97b","Type":"ContainerDied","Data":"12227c7eed281f6b42957f062ad78b80d6d2b03728a4c3fe5a763ea19ee92f60"} Jul 31 14:22:37.649939 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:37.649896 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hr9hd"] Jul 31 14:22:37.652266 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:22:37.652238 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod532f707f_36b4_466f_8dce_e81bff34777e.slice/crio-9496c6933d887268c3ac63c7068cc8f6c4c9f144424550be3220a553d16ee6f1 WatchSource:0}: Error finding container 9496c6933d887268c3ac63c7068cc8f6c4c9f144424550be3220a553d16ee6f1: Status 404 returned error can't find the container with id 9496c6933d887268c3ac63c7068cc8f6c4c9f144424550be3220a553d16ee6f1 Jul 31 14:22:38.143241 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.143201 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" event={"ID":"08612026-960a-44c1-9638-0f0e87a9d148","Type":"ContainerStarted","Data":"12497beb4bceaf2f5f6b60c7c95791e44666da204993e2a9a4fa9fe91f965461"} Jul 31 14:22:38.144657 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.144630 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" event={"ID":"532f707f-36b4-466f-8dce-e81bff34777e","Type":"ContainerStarted","Data":"f1b6b1d591ddd5f35cf5e69005420fc41211e7ed767fa630f6e4b91ed3193872"} Jul 31 14:22:38.144758 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.144672 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" event={"ID":"532f707f-36b4-466f-8dce-e81bff34777e","Type":"ContainerStarted","Data":"9496c6933d887268c3ac63c7068cc8f6c4c9f144424550be3220a553d16ee6f1"} Jul 31 14:22:38.160180 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.160130 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4fpdx" podStartSLOduration=1.970531196 podStartE2EDuration="6.160115966s" totalImagesPullingTime="4.18958477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:22:38.158296357 +0000 UTC m=+623.666675424" watchObservedRunningTime="2026-07-31 14:22:38.160115966 +0000 UTC m=+623.668495031" Jul 31 14:22:38.173784 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.173721 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" podStartSLOduration=2.173699764 podStartE2EDuration="2.173699764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:22:38.172657842 +0000 UTC m=+623.681036910" watchObservedRunningTime="2026-07-31 14:22:38.173699764 +0000 UTC m=+623.682078914" Jul 31 14:22:38.275101 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.275075 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:38.331732 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.331686 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:38.434089 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.434004 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-bundle\") pod \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " Jul 31 14:22:38.434089 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.434055 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-util\") pod \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " Jul 31 14:22:38.434089 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.434080 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkvk9\" (UniqueName: \"kubernetes.io/projected/08b8c271-5548-4b1b-9a12-b2f26e44f97b-kube-api-access-bkvk9\") pod \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\" (UID: \"08b8c271-5548-4b1b-9a12-b2f26e44f97b\") " Jul 31 14:22:38.434505 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.434474 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-bundle" (OuterVolumeSpecName: "bundle") pod "08b8c271-5548-4b1b-9a12-b2f26e44f97b" (UID: "08b8c271-5548-4b1b-9a12-b2f26e44f97b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:22:38.436199 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.436174 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b8c271-5548-4b1b-9a12-b2f26e44f97b-kube-api-access-bkvk9" (OuterVolumeSpecName: "kube-api-access-bkvk9") pod "08b8c271-5548-4b1b-9a12-b2f26e44f97b" (UID: "08b8c271-5548-4b1b-9a12-b2f26e44f97b"). InnerVolumeSpecName "kube-api-access-bkvk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:22:38.439281 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.439256 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-util" (OuterVolumeSpecName: "util") pod "08b8c271-5548-4b1b-9a12-b2f26e44f97b" (UID: "08b8c271-5548-4b1b-9a12-b2f26e44f97b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:22:38.534693 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.534660 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:22:38.534693 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.534688 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b8c271-5548-4b1b-9a12-b2f26e44f97b-util\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:22:38.534693 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:38.534697 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkvk9\" (UniqueName: \"kubernetes.io/projected/08b8c271-5548-4b1b-9a12-b2f26e44f97b-kube-api-access-bkvk9\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:22:39.150196 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:39.150157 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" event={"ID":"08b8c271-5548-4b1b-9a12-b2f26e44f97b","Type":"ContainerDied","Data":"643e37d01a8433ce73c67a127bd4c689e58e3a62c39eec796d96cbed04ecfe86"} Jul 31 14:22:39.150196 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:39.150196 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7b28p" Jul 31 14:22:39.150615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:39.150210 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643e37d01a8433ce73c67a127bd4c689e58e3a62c39eec796d96cbed04ecfe86" Jul 31 14:22:39.150615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:39.150349 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:39.714339 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:39.714306 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:42.488289 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:42.488237 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:42.583715 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:42.583681 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:45.157036 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:45.157006 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hr9hd" Jul 31 14:22:57.036843 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:57.036807 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:57.090726 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:57.090687 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:22:58.982803 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:22:58.982765 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:00.663607 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:00.663572 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:00.764160 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:00.764122 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:02.312999 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.312957 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp"] Jul 31 14:23:02.313404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.313250 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerName="extract" Jul 31 14:23:02.313404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.313262 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerName="extract" Jul 31 14:23:02.313404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.313280 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerName="util" Jul 31 14:23:02.313404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.313286 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerName="util" Jul 31 14:23:02.313404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.313293 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerName="pull" Jul 31 14:23:02.313404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.313298 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerName="pull" Jul 31 14:23:02.313404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.313346 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b8c271-5548-4b1b-9a12-b2f26e44f97b" containerName="extract" Jul 31 14:23:02.316424 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.316407 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.319122 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.319099 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 14:23:02.319227 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.319099 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 14:23:02.320510 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.320490 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fdsgp\"" Jul 31 14:23:02.324040 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.324020 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp"] Jul 31 14:23:02.414853 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.414816 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjr4m\" (UniqueName: \"kubernetes.io/projected/99040abb-7544-46af-ac72-5b98064bf379-kube-api-access-wjr4m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.415033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.414885 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.415033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.414929 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.433404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.433341 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:02.516278 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.516239 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.516460 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.516282 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.516460 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.516352 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjr4m\" (UniqueName: \"kubernetes.io/projected/99040abb-7544-46af-ac72-5b98064bf379-kube-api-access-wjr4m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.516630 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.516610 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.516689 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.516635 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.525213 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.525178 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjr4m\" (UniqueName: \"kubernetes.io/projected/99040abb-7544-46af-ac72-5b98064bf379-kube-api-access-wjr4m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.626658 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.626629 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:02.755210 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:02.755182 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp"] Jul 31 14:23:02.757626 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:23:02.757592 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99040abb_7544_46af_ac72_5b98064bf379.slice/crio-b8d39f4ca452d67a2b2a975d9818e41b0004d154774c1d820d3ce8f7362a622b WatchSource:0}: Error finding container b8d39f4ca452d67a2b2a975d9818e41b0004d154774c1d820d3ce8f7362a622b: Status 404 returned error can't find the container with id b8d39f4ca452d67a2b2a975d9818e41b0004d154774c1d820d3ce8f7362a622b Jul 31 14:23:03.246159 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:03.246126 2558 generic.go:356] "Generic (PLEG): container finished" podID="99040abb-7544-46af-ac72-5b98064bf379" containerID="726ca3c8d8014f8f97c0d3499d016a36f585d352434692cf7b42e6eb5feb842a" exitCode=0 Jul 31 14:23:03.246334 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:03.246208 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" event={"ID":"99040abb-7544-46af-ac72-5b98064bf379","Type":"ContainerDied","Data":"726ca3c8d8014f8f97c0d3499d016a36f585d352434692cf7b42e6eb5feb842a"} Jul 31 14:23:03.246334 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:03.246256 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" event={"ID":"99040abb-7544-46af-ac72-5b98064bf379","Type":"ContainerStarted","Data":"b8d39f4ca452d67a2b2a975d9818e41b0004d154774c1d820d3ce8f7362a622b"} Jul 31 14:23:03.246638 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:03.246624 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:23:04.252427 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:04.252340 2558 generic.go:356] "Generic (PLEG): container finished" podID="99040abb-7544-46af-ac72-5b98064bf379" containerID="7de5b1a9bd3d8ba86a89486a823123e4ad9d85e4ab358cca7c851dcd7402c81b" exitCode=0 Jul 31 14:23:04.252778 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:04.252430 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" event={"ID":"99040abb-7544-46af-ac72-5b98064bf379","Type":"ContainerDied","Data":"7de5b1a9bd3d8ba86a89486a823123e4ad9d85e4ab358cca7c851dcd7402c81b"} Jul 31 14:23:05.259173 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:05.259134 2558 generic.go:356] "Generic (PLEG): container finished" podID="99040abb-7544-46af-ac72-5b98064bf379" containerID="e1fe994876d54b513edf6d4add947fdabebe4ee3fa22fef17ff0213a47eebaba" exitCode=0 Jul 31 14:23:05.259555 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:05.259185 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" event={"ID":"99040abb-7544-46af-ac72-5b98064bf379","Type":"ContainerDied","Data":"e1fe994876d54b513edf6d4add947fdabebe4ee3fa22fef17ff0213a47eebaba"} Jul 31 14:23:06.386102 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.386075 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:06.450576 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.450541 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjr4m\" (UniqueName: \"kubernetes.io/projected/99040abb-7544-46af-ac72-5b98064bf379-kube-api-access-wjr4m\") pod \"99040abb-7544-46af-ac72-5b98064bf379\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " Jul 31 14:23:06.450759 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.450626 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-bundle\") pod \"99040abb-7544-46af-ac72-5b98064bf379\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " Jul 31 14:23:06.450759 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.450650 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-util\") pod \"99040abb-7544-46af-ac72-5b98064bf379\" (UID: \"99040abb-7544-46af-ac72-5b98064bf379\") " Jul 31 14:23:06.451320 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.451287 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-bundle" (OuterVolumeSpecName: "bundle") pod "99040abb-7544-46af-ac72-5b98064bf379" (UID: "99040abb-7544-46af-ac72-5b98064bf379"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:23:06.452652 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.452626 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99040abb-7544-46af-ac72-5b98064bf379-kube-api-access-wjr4m" (OuterVolumeSpecName: "kube-api-access-wjr4m") pod "99040abb-7544-46af-ac72-5b98064bf379" (UID: "99040abb-7544-46af-ac72-5b98064bf379"). InnerVolumeSpecName "kube-api-access-wjr4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:23:06.456583 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.456544 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-util" (OuterVolumeSpecName: "util") pod "99040abb-7544-46af-ac72-5b98064bf379" (UID: "99040abb-7544-46af-ac72-5b98064bf379"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:23:06.551784 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.551713 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjr4m\" (UniqueName: \"kubernetes.io/projected/99040abb-7544-46af-ac72-5b98064bf379-kube-api-access-wjr4m\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:23:06.551784 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.551737 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:23:06.551784 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.551745 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99040abb-7544-46af-ac72-5b98064bf379-util\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:23:06.608695 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.608655 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:06.656654 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.656618 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb"] Jul 31 14:23:06.656929 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.656916 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99040abb-7544-46af-ac72-5b98064bf379" containerName="extract" Jul 31 14:23:06.656984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.656933 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99040abb-7544-46af-ac72-5b98064bf379" containerName="extract" Jul 31 14:23:06.656984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.656948 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99040abb-7544-46af-ac72-5b98064bf379" containerName="pull" Jul 31 14:23:06.656984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.656954 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99040abb-7544-46af-ac72-5b98064bf379" containerName="pull" Jul 31 14:23:06.656984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.656966 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99040abb-7544-46af-ac72-5b98064bf379" containerName="util" Jul 31 14:23:06.656984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.656971 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99040abb-7544-46af-ac72-5b98064bf379" containerName="util" Jul 31 14:23:06.657133 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.657021 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="99040abb-7544-46af-ac72-5b98064bf379" containerName="extract" Jul 31 14:23:06.660009 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.659991 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.662811 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.662755 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 31 14:23:06.662958 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.662827 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 31 14:23:06.663014 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.662982 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 31 14:23:06.663132 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.663116 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 31 14:23:06.663224 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.663209 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 31 14:23:06.663303 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.663289 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4jnpn\"" Jul 31 14:23:06.673695 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.673676 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb"] Jul 31 14:23:06.680153 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.680114 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:06.753678 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.753648 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmzf6\" (UniqueName: \"kubernetes.io/projected/77394d7d-b35b-4bb9-b391-fa9e60a73588-kube-api-access-qmzf6\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.753827 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.753683 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77394d7d-b35b-4bb9-b391-fa9e60a73588-cert\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.753827 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.753721 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77394d7d-b35b-4bb9-b391-fa9e60a73588-manager-config\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.753827 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.753744 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77394d7d-b35b-4bb9-b391-fa9e60a73588-metrics-cert\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.854484 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.854403 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77394d7d-b35b-4bb9-b391-fa9e60a73588-manager-config\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.854484 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.854437 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77394d7d-b35b-4bb9-b391-fa9e60a73588-metrics-cert\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.854712 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.854488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzf6\" (UniqueName: \"kubernetes.io/projected/77394d7d-b35b-4bb9-b391-fa9e60a73588-kube-api-access-qmzf6\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.854712 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.854506 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77394d7d-b35b-4bb9-b391-fa9e60a73588-cert\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.855110 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.855081 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77394d7d-b35b-4bb9-b391-fa9e60a73588-manager-config\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.856824 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.856808 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77394d7d-b35b-4bb9-b391-fa9e60a73588-metrics-cert\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.856885 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.856850 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77394d7d-b35b-4bb9-b391-fa9e60a73588-cert\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.864058 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.864037 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzf6\" (UniqueName: \"kubernetes.io/projected/77394d7d-b35b-4bb9-b391-fa9e60a73588-kube-api-access-qmzf6\") pod \"lws-controller-manager-58f6dff6d5-rdmpb\" (UID: \"77394d7d-b35b-4bb9-b391-fa9e60a73588\") " pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:06.968738 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:06.968696 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:07.112711 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:07.112486 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb"] Jul 31 14:23:07.115929 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:23:07.115881 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77394d7d_b35b_4bb9_b391_fa9e60a73588.slice/crio-b5a1c899edaea184b427511094deb39461ee0d52925c8a06b4b31f5ae0b5185b WatchSource:0}: Error finding container b5a1c899edaea184b427511094deb39461ee0d52925c8a06b4b31f5ae0b5185b: Status 404 returned error can't find the container with id b5a1c899edaea184b427511094deb39461ee0d52925c8a06b4b31f5ae0b5185b Jul 31 14:23:07.270962 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:07.270933 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" Jul 31 14:23:07.271126 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:07.270936 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5m6qp" event={"ID":"99040abb-7544-46af-ac72-5b98064bf379","Type":"ContainerDied","Data":"b8d39f4ca452d67a2b2a975d9818e41b0004d154774c1d820d3ce8f7362a622b"} Jul 31 14:23:07.271126 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:07.271099 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8d39f4ca452d67a2b2a975d9818e41b0004d154774c1d820d3ce8f7362a622b" Jul 31 14:23:07.272397 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:07.272370 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" event={"ID":"77394d7d-b35b-4bb9-b391-fa9e60a73588","Type":"ContainerStarted","Data":"b5a1c899edaea184b427511094deb39461ee0d52925c8a06b4b31f5ae0b5185b"} Jul 31 14:23:15.456715 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:15.456673 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:15.546822 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:15.546790 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:16.053114 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.053075 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:16.255777 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.255739 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:16.355210 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.355123 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:16.400406 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.400371 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2"] Jul 31 14:23:16.403673 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.403656 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.407146 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.407127 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2k6vh\"" Jul 31 14:23:16.407756 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.407737 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 31 14:23:16.408033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.408016 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 31 14:23:16.408127 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.408033 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 31 14:23:16.408227 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.408214 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 31 14:23:16.420279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.420252 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2"] Jul 31 14:23:16.502599 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.502573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:16.536232 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.536206 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44a0597b-822f-4b83-be89-95054f320116-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.536323 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.536246 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44a0597b-822f-4b83-be89-95054f320116-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.536366 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.536327 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44a0597b-822f-4b83-be89-95054f320116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.536366 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.536356 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44a0597b-822f-4b83-be89-95054f320116-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.536441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.536407 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqh4\" (UniqueName: \"kubernetes.io/projected/44a0597b-822f-4b83-be89-95054f320116-kube-api-access-ztqh4\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.612310 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.612243 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:16.637360 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.637331 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44a0597b-822f-4b83-be89-95054f320116-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.637472 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.637372 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44a0597b-822f-4b83-be89-95054f320116-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.637472 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.637414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44a0597b-822f-4b83-be89-95054f320116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.637472 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.637440 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44a0597b-822f-4b83-be89-95054f320116-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.637624 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.637495 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqh4\" (UniqueName: \"kubernetes.io/projected/44a0597b-822f-4b83-be89-95054f320116-kube-api-access-ztqh4\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.637752 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.637727 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44a0597b-822f-4b83-be89-95054f320116-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.639716 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.639696 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44a0597b-822f-4b83-be89-95054f320116-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.639849 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.639833 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44a0597b-822f-4b83-be89-95054f320116-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.639952 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.639936 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44a0597b-822f-4b83-be89-95054f320116-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.652049 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.652025 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqh4\" (UniqueName: \"kubernetes.io/projected/44a0597b-822f-4b83-be89-95054f320116-kube-api-access-ztqh4\") pod \"opendatahub-operator-controller-manager-698c94c45b-5d4h2\" (UID: \"44a0597b-822f-4b83-be89-95054f320116\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.707978 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.707948 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:16.713515 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.713494 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:16.805517 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.805477 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:16.848665 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.848641 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2"] Jul 31 14:23:16.850402 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:23:16.850374 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a0597b_822f_4b83_be89_95054f320116.slice/crio-6a616e50af9edf5866e51a72a7361bd96c19812ce57f16aa94e5cb1e36347766 WatchSource:0}: Error finding container 6a616e50af9edf5866e51a72a7361bd96c19812ce57f16aa94e5cb1e36347766: Status 404 returned error can't find the container with id 6a616e50af9edf5866e51a72a7361bd96c19812ce57f16aa94e5cb1e36347766 Jul 31 14:23:16.905975 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:16.905942 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:17.005063 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:17.005031 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:17.313303 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:17.313213 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" event={"ID":"44a0597b-822f-4b83-be89-95054f320116","Type":"ContainerStarted","Data":"6a616e50af9edf5866e51a72a7361bd96c19812ce57f16aa94e5cb1e36347766"} Jul 31 14:23:17.999898 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:23:17.999765 2558 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/leader-worker-set/lws-rhel9@sha256=70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f/signature-1: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f" Jul 31 14:23:18.000373 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:23:17.999922 2558 kuberuntime_manager.go:1459] "Unhandled Error" err="container manager start failed in pod lws-controller-manager-58f6dff6d5-rdmpb_openshift-lws-operator(77394d7d-b35b-4bb9-b391-fa9e60a73588): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/leader-worker-set/lws-rhel9@sha256=70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f/signature-1: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 31 14:23:18.000373 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:23:17.999973 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/leader-worker-set/lws-rhel9@sha256=70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f/signature-1: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" podUID="77394d7d-b35b-4bb9-b391-fa9e60a73588" Jul 31 14:23:18.323231 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:23:18.323192 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/leader-worker-set/lws-rhel9@sha256=70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f/signature-1: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" podUID="77394d7d-b35b-4bb9-b391-fa9e60a73588" Jul 31 14:23:20.331612 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:20.331570 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" event={"ID":"44a0597b-822f-4b83-be89-95054f320116","Type":"ContainerStarted","Data":"fcbfb4eeeaa38235da917bede29377f4c802b3f2341080e82c5559b72ddaed5b"} Jul 31 14:23:21.337750 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:21.337715 2558 generic.go:356] "Generic (PLEG): container finished" podID="44a0597b-822f-4b83-be89-95054f320116" containerID="fcbfb4eeeaa38235da917bede29377f4c802b3f2341080e82c5559b72ddaed5b" exitCode=0 Jul 31 14:23:21.338229 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:21.337803 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" event={"ID":"44a0597b-822f-4b83-be89-95054f320116","Type":"ContainerDied","Data":"fcbfb4eeeaa38235da917bede29377f4c802b3f2341080e82c5559b72ddaed5b"} Jul 31 14:23:22.344349 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:22.344316 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" event={"ID":"44a0597b-822f-4b83-be89-95054f320116","Type":"ContainerStarted","Data":"4c09a745cd1daa961edb58979bfd09371bc57f9c2aa6db3233cccbb4f27fa4b0"} Jul 31 14:23:22.344720 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:22.344483 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:22.364896 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:22.364852 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" podStartSLOduration=2.078798372 podStartE2EDuration="6.364837167s" totalImagesPullingTime="3.208867028s" totalInitContainerRuntime="1.077171767s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:22.363179193 +0000 UTC m=+667.871558260" watchObservedRunningTime="2026-07-31 14:23:22.364837167 +0000 UTC m=+667.873216234" Jul 31 14:23:23.245440 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:23.245400 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:23.300211 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:23.300178 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:28.469184 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:28.469144 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:28.566582 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:28.566548 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:33.350976 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:33.350945 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-5d4h2" Jul 31 14:23:35.570988 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:35.570956 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:35.680167 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:35.680130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:36.289074 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:36.289043 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:36.349310 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:36.349279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:36.411330 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:36.411298 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" event={"ID":"77394d7d-b35b-4bb9-b391-fa9e60a73588","Type":"ContainerStarted","Data":"11020aaa036a18f17df9be8a675d4e4d165760bab1c2a042139088a9973c1eca"} Jul 31 14:23:36.411532 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:36.411511 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:36.430815 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:36.430763 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" podStartSLOduration=2.052470077 podStartE2EDuration="30.430746661s" totalImagesPullingTime="28.378276584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-31 14:23:36.429323359 +0000 UTC m=+681.937702426" watchObservedRunningTime="2026-07-31 14:23:36.430746661 +0000 UTC m=+681.939125729" Jul 31 14:23:36.433817 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:36.433788 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:36.755900 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:36.755864 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:36.836081 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:36.836035 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:37.186872 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:37.186835 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:38.818241 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:38.818207 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:38.902407 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:38.902378 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj"] Jul 31 14:23:38.917702 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:38.917675 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj"] Jul 31 14:23:38.917846 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:38.917770 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:38.920507 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:38.920482 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 31 14:23:38.920666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:38.920531 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 31 14:23:38.920666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:38.920568 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mmkg9\"" Jul 31 14:23:39.022821 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.022786 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlcs\" (UniqueName: \"kubernetes.io/projected/cb68771c-54df-4f39-9822-851485d40255-kube-api-access-pqlcs\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.023056 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.022845 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb68771c-54df-4f39-9822-851485d40255-tls-certs\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.023056 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.022956 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb68771c-54df-4f39-9822-851485d40255-tmp\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.123675 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.123640 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb68771c-54df-4f39-9822-851485d40255-tls-certs\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.123844 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.123708 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb68771c-54df-4f39-9822-851485d40255-tmp\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.123844 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.123761 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlcs\" (UniqueName: \"kubernetes.io/projected/cb68771c-54df-4f39-9822-851485d40255-kube-api-access-pqlcs\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.123844 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:23:39.123800 2558 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 31 14:23:39.124005 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:23:39.123867 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb68771c-54df-4f39-9822-851485d40255-tls-certs podName:cb68771c-54df-4f39-9822-851485d40255 nodeName:}" failed. No retries permitted until 2026-07-31 14:23:39.623845962 +0000 UTC m=+685.132225022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cb68771c-54df-4f39-9822-851485d40255-tls-certs") pod "kube-auth-proxy-7c8b44b796-w7hcj" (UID: "cb68771c-54df-4f39-9822-851485d40255") : secret "kube-auth-proxy-tls" not found Jul 31 14:23:39.126005 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.125985 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb68771c-54df-4f39-9822-851485d40255-tmp\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.132846 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.132821 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlcs\" (UniqueName: \"kubernetes.io/projected/cb68771c-54df-4f39-9822-851485d40255-kube-api-access-pqlcs\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.508511 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.508428 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:39.628019 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.627987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb68771c-54df-4f39-9822-851485d40255-tls-certs\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.630449 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.630423 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb68771c-54df-4f39-9822-851485d40255-tls-certs\") pod \"kube-auth-proxy-7c8b44b796-w7hcj\" (UID: \"cb68771c-54df-4f39-9822-851485d40255\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.830353 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.830315 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" Jul 31 14:23:39.849278 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.849249 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:39.947672 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.947634 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:39.968142 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:39.968089 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj"] Jul 31 14:23:39.970558 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:23:39.970534 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb68771c_54df_4f39_9822_851485d40255.slice/crio-ec77d2409541131f233ff66a58c3820982398655d33d5a28e373c39033047126 WatchSource:0}: Error finding container ec77d2409541131f233ff66a58c3820982398655d33d5a28e373c39033047126: Status 404 returned error can't find the container with id ec77d2409541131f233ff66a58c3820982398655d33d5a28e373c39033047126 Jul 31 14:23:40.432233 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:40.432193 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" event={"ID":"cb68771c-54df-4f39-9822-851485d40255","Type":"ContainerStarted","Data":"ec77d2409541131f233ff66a58c3820982398655d33d5a28e373c39033047126"} Jul 31 14:23:40.800545 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:40.800446 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:43.447967 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:43.447929 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" event={"ID":"cb68771c-54df-4f39-9822-851485d40255","Type":"ContainerStarted","Data":"9e50951b692dc4f6b06b51f9605f8c9cf2eb30aff57b97b2a63e78ab96bb8c9b"} Jul 31 14:23:43.472492 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:43.472443 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-w7hcj" podStartSLOduration=2.602840848 podStartE2EDuration="5.472429274s" totalImagesPullingTime="2.869588426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:43.470874412 +0000 UTC m=+688.979253478" watchObservedRunningTime="2026-07-31 14:23:43.472429274 +0000 UTC m=+688.980808339" Jul 31 14:23:46.100280 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:46.100232 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:46.209578 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:46.209529 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:46.360179 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:46.360101 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:46.468414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:46.468356 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:46.635647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:46.635610 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:46.747442 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:46.747402 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:46.841290 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:46.841251 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:47.050363 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:47.050271 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:23:47.417943 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:47.417896 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58f6dff6d5-rdmpb" Jul 31 14:23:50.580594 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:23:50.580555 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:01.674100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:01.674064 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:37.372239 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:37.372198 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:37.466895 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:37.466853 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:40.206800 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.206755 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67"] Jul 31 14:24:40.210262 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.210238 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.213420 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.213397 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 31 14:24:40.214741 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.214719 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 31 14:24:40.214823 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.214721 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fdsgp\"" Jul 31 14:24:40.219665 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.219640 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67"] Jul 31 14:24:40.224760 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.224735 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.224844 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.224782 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.224914 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.224883 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlg8q\" (UniqueName: \"kubernetes.io/projected/2978f7d9-f54f-48f1-9708-c528e4db68e1-kube-api-access-jlg8q\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.306710 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.306675 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249"] Jul 31 14:24:40.310301 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.310280 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.319070 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.319049 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249"] Jul 31 14:24:40.325280 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.325255 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlg8q\" (UniqueName: \"kubernetes.io/projected/2978f7d9-f54f-48f1-9708-c528e4db68e1-kube-api-access-jlg8q\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.325386 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.325297 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.325386 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.325328 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.325386 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.325361 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.325584 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.325392 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.325584 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.325447 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmdd\" (UniqueName: \"kubernetes.io/projected/5f21c2db-936e-455b-9750-61b10c90acd1-kube-api-access-qqmdd\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.325696 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.325675 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.325757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.325709 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.335577 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.335537 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlg8q\" (UniqueName: \"kubernetes.io/projected/2978f7d9-f54f-48f1-9708-c528e4db68e1-kube-api-access-jlg8q\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.400027 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.399990 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2"] Jul 31 14:24:40.403546 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.403520 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.412685 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.412659 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2"] Jul 31 14:24:40.426463 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.426434 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.426616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.426477 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.426616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.426503 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.426616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.426533 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmdd\" (UniqueName: \"kubernetes.io/projected/5f21c2db-936e-455b-9750-61b10c90acd1-kube-api-access-qqmdd\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.426616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.426555 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmj5\" (UniqueName: \"kubernetes.io/projected/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-kube-api-access-dzmj5\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.426616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.426581 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.426867 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.426797 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.426867 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.426828 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.435948 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.435890 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmdd\" (UniqueName: \"kubernetes.io/projected/5f21c2db-936e-455b-9750-61b10c90acd1-kube-api-access-qqmdd\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.505540 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.505440 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9"] Jul 31 14:24:40.511640 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.511610 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.515518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.515487 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9"] Jul 31 14:24:40.519729 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.519700 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:40.527100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.527071 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmj5\" (UniqueName: \"kubernetes.io/projected/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-kube-api-access-dzmj5\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.527221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.527113 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.527221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.527146 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.527221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.527185 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9x4\" (UniqueName: \"kubernetes.io/projected/23033390-d6eb-439a-a8dd-41925c4e1864-kube-api-access-4l9x4\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.527221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.527215 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.527457 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.527256 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.527610 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.527587 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.527746 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.527621 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.536037 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.536008 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmj5\" (UniqueName: \"kubernetes.io/projected/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-kube-api-access-dzmj5\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.620279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.620229 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:40.628464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.628352 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.628464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.628430 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9x4\" (UniqueName: \"kubernetes.io/projected/23033390-d6eb-439a-a8dd-41925c4e1864-kube-api-access-4l9x4\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.628464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.628465 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.628865 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.628830 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.628865 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.628830 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.638655 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.638626 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9x4\" (UniqueName: \"kubernetes.io/projected/23033390-d6eb-439a-a8dd-41925c4e1864-kube-api-access-4l9x4\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.656953 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.656922 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67"] Jul 31 14:24:40.702511 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.702442 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" event={"ID":"2978f7d9-f54f-48f1-9708-c528e4db68e1","Type":"ContainerStarted","Data":"d09c272f6a1a383fc33b01ddd685e401e38c7c068848eaa67fc47beef152d70c"} Jul 31 14:24:40.714484 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.714453 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:40.777414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.777380 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249"] Jul 31 14:24:40.779882 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:24:40.779854 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f21c2db_936e_455b_9750_61b10c90acd1.slice/crio-b5638a4887d932086f14ec79a70c500b5215c959cf75c2a688f8b510c20d0981 WatchSource:0}: Error finding container b5638a4887d932086f14ec79a70c500b5215c959cf75c2a688f8b510c20d0981: Status 404 returned error can't find the container with id b5638a4887d932086f14ec79a70c500b5215c959cf75c2a688f8b510c20d0981 Jul 31 14:24:40.822815 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.822788 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:40.873069 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.873043 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2"] Jul 31 14:24:40.877837 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:24:40.877657 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b69f7c_d52e_48f4_8ad9_9c9d74b08528.slice/crio-20aa3f537837709cd7e43f089fe167f66a1eb456e4c673e1a4314bed504b57a7 WatchSource:0}: Error finding container 20aa3f537837709cd7e43f089fe167f66a1eb456e4c673e1a4314bed504b57a7: Status 404 returned error can't find the container with id 20aa3f537837709cd7e43f089fe167f66a1eb456e4c673e1a4314bed504b57a7 Jul 31 14:24:40.963872 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:40.963848 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9"] Jul 31 14:24:40.998777 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:24:40.998748 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23033390_d6eb_439a_a8dd_41925c4e1864.slice/crio-7b359c96b966f18528dc9a8717ab44dad6473e1e227697b9c3ef6c187aa1e0df WatchSource:0}: Error finding container 7b359c96b966f18528dc9a8717ab44dad6473e1e227697b9c3ef6c187aa1e0df: Status 404 returned error can't find the container with id 7b359c96b966f18528dc9a8717ab44dad6473e1e227697b9c3ef6c187aa1e0df Jul 31 14:24:41.708656 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.708616 2558 generic.go:356] "Generic (PLEG): container finished" podID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerID="bc0387fd074d4f37a2c3098e3ead5034e28b0b10f99dc2e01198eca452c41b2a" exitCode=0 Jul 31 14:24:41.709165 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.708702 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" event={"ID":"2978f7d9-f54f-48f1-9708-c528e4db68e1","Type":"ContainerDied","Data":"bc0387fd074d4f37a2c3098e3ead5034e28b0b10f99dc2e01198eca452c41b2a"} Jul 31 14:24:41.710359 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.710319 2558 generic.go:356] "Generic (PLEG): container finished" podID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerID="c387a98867c3550b258676b5a679a31be9cc2303f230c8f67d368f89df8cf105" exitCode=0 Jul 31 14:24:41.710443 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.710361 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" event={"ID":"31b69f7c-d52e-48f4-8ad9-9c9d74b08528","Type":"ContainerDied","Data":"c387a98867c3550b258676b5a679a31be9cc2303f230c8f67d368f89df8cf105"} Jul 31 14:24:41.710443 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.710412 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" event={"ID":"31b69f7c-d52e-48f4-8ad9-9c9d74b08528","Type":"ContainerStarted","Data":"20aa3f537837709cd7e43f089fe167f66a1eb456e4c673e1a4314bed504b57a7"} Jul 31 14:24:41.711984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.711961 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f21c2db-936e-455b-9750-61b10c90acd1" containerID="8718585a8cc08521a8a8f386ac290a75e9d85d001c3eb9ea0ea8f4d9056381e9" exitCode=0 Jul 31 14:24:41.712209 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.711989 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" event={"ID":"5f21c2db-936e-455b-9750-61b10c90acd1","Type":"ContainerDied","Data":"8718585a8cc08521a8a8f386ac290a75e9d85d001c3eb9ea0ea8f4d9056381e9"} Jul 31 14:24:41.712209 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.712029 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" event={"ID":"5f21c2db-936e-455b-9750-61b10c90acd1","Type":"ContainerStarted","Data":"b5638a4887d932086f14ec79a70c500b5215c959cf75c2a688f8b510c20d0981"} Jul 31 14:24:41.715204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.714946 2558 generic.go:356] "Generic (PLEG): container finished" podID="23033390-d6eb-439a-a8dd-41925c4e1864" containerID="b311dca832035d36c445dacc7cbd88839e72dca80b2ddcffb57fa63b27c4f75b" exitCode=0 Jul 31 14:24:41.715204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.714986 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" event={"ID":"23033390-d6eb-439a-a8dd-41925c4e1864","Type":"ContainerDied","Data":"b311dca832035d36c445dacc7cbd88839e72dca80b2ddcffb57fa63b27c4f75b"} Jul 31 14:24:41.715204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:41.715032 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" event={"ID":"23033390-d6eb-439a-a8dd-41925c4e1864","Type":"ContainerStarted","Data":"7b359c96b966f18528dc9a8717ab44dad6473e1e227697b9c3ef6c187aa1e0df"} Jul 31 14:24:42.721647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:42.721605 2558 generic.go:356] "Generic (PLEG): container finished" podID="23033390-d6eb-439a-a8dd-41925c4e1864" containerID="40d50c793f1509de29bb4e659c425633ace2241f34a7eae58608ef8478844c83" exitCode=0 Jul 31 14:24:42.722008 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:42.721672 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" event={"ID":"23033390-d6eb-439a-a8dd-41925c4e1864","Type":"ContainerDied","Data":"40d50c793f1509de29bb4e659c425633ace2241f34a7eae58608ef8478844c83"} Jul 31 14:24:43.728194 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:43.728162 2558 generic.go:356] "Generic (PLEG): container finished" podID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerID="7ffc7e742f013c94826afc9a4d42b309979705cc2f2cf640ba742c8e794badbd" exitCode=0 Jul 31 14:24:43.728601 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:43.728251 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" event={"ID":"31b69f7c-d52e-48f4-8ad9-9c9d74b08528","Type":"ContainerDied","Data":"7ffc7e742f013c94826afc9a4d42b309979705cc2f2cf640ba742c8e794badbd"} Jul 31 14:24:43.730150 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:43.730130 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f21c2db-936e-455b-9750-61b10c90acd1" containerID="c7bdf90825e9c710860c67ef949bc298d8111107163cf01daf9e942f76738d69" exitCode=0 Jul 31 14:24:43.730238 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:43.730187 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" event={"ID":"5f21c2db-936e-455b-9750-61b10c90acd1","Type":"ContainerDied","Data":"c7bdf90825e9c710860c67ef949bc298d8111107163cf01daf9e942f76738d69"} Jul 31 14:24:43.732464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:43.732440 2558 generic.go:356] "Generic (PLEG): container finished" podID="23033390-d6eb-439a-a8dd-41925c4e1864" containerID="2b6a41547793cfee688848b69346592edaf8f1910d61109452ccc4b200cbb250" exitCode=0 Jul 31 14:24:43.732563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:43.732510 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" event={"ID":"23033390-d6eb-439a-a8dd-41925c4e1864","Type":"ContainerDied","Data":"2b6a41547793cfee688848b69346592edaf8f1910d61109452ccc4b200cbb250"} Jul 31 14:24:43.734497 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:43.734479 2558 generic.go:356] "Generic (PLEG): container finished" podID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerID="a6f06e87b52c1e5fb44935aea6e40dc14c5c8e112937fdb9833373a4b2b179c9" exitCode=0 Jul 31 14:24:43.734561 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:43.734541 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" event={"ID":"2978f7d9-f54f-48f1-9708-c528e4db68e1","Type":"ContainerDied","Data":"a6f06e87b52c1e5fb44935aea6e40dc14c5c8e112937fdb9833373a4b2b179c9"} Jul 31 14:24:44.741846 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.741808 2558 generic.go:356] "Generic (PLEG): container finished" podID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerID="f8220fd3451a875a9d5e886be9fa7e570442e72cd8df5983bc67dc4d460dabc9" exitCode=0 Jul 31 14:24:44.742311 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.741896 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" event={"ID":"2978f7d9-f54f-48f1-9708-c528e4db68e1","Type":"ContainerDied","Data":"f8220fd3451a875a9d5e886be9fa7e570442e72cd8df5983bc67dc4d460dabc9"} Jul 31 14:24:44.743963 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.743940 2558 generic.go:356] "Generic (PLEG): container finished" podID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerID="ac5d6fb55de591d5332a79fdf716a46eb6e4401cc7106fa707515669e339c23f" exitCode=0 Jul 31 14:24:44.744069 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.744018 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" event={"ID":"31b69f7c-d52e-48f4-8ad9-9c9d74b08528","Type":"ContainerDied","Data":"ac5d6fb55de591d5332a79fdf716a46eb6e4401cc7106fa707515669e339c23f"} Jul 31 14:24:44.745883 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.745860 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f21c2db-936e-455b-9750-61b10c90acd1" containerID="effe35dd16cb0314a9d1c1af77be709e9c6c96c54142d7c92db53630699113b6" exitCode=0 Jul 31 14:24:44.745992 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.745887 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" event={"ID":"5f21c2db-936e-455b-9750-61b10c90acd1","Type":"ContainerDied","Data":"effe35dd16cb0314a9d1c1af77be709e9c6c96c54142d7c92db53630699113b6"} Jul 31 14:24:44.875965 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.875940 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:44.966220 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.966180 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-bundle\") pod \"23033390-d6eb-439a-a8dd-41925c4e1864\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " Jul 31 14:24:44.966416 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.966229 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l9x4\" (UniqueName: \"kubernetes.io/projected/23033390-d6eb-439a-a8dd-41925c4e1864-kube-api-access-4l9x4\") pod \"23033390-d6eb-439a-a8dd-41925c4e1864\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " Jul 31 14:24:44.966416 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.966252 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-util\") pod \"23033390-d6eb-439a-a8dd-41925c4e1864\" (UID: \"23033390-d6eb-439a-a8dd-41925c4e1864\") " Jul 31 14:24:44.966748 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.966710 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-bundle" (OuterVolumeSpecName: "bundle") pod "23033390-d6eb-439a-a8dd-41925c4e1864" (UID: "23033390-d6eb-439a-a8dd-41925c4e1864"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:24:44.968327 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.968302 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23033390-d6eb-439a-a8dd-41925c4e1864-kube-api-access-4l9x4" (OuterVolumeSpecName: "kube-api-access-4l9x4") pod "23033390-d6eb-439a-a8dd-41925c4e1864" (UID: "23033390-d6eb-439a-a8dd-41925c4e1864"). InnerVolumeSpecName "kube-api-access-4l9x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:24:44.971564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:44.971542 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-util" (OuterVolumeSpecName: "util") pod "23033390-d6eb-439a-a8dd-41925c4e1864" (UID: "23033390-d6eb-439a-a8dd-41925c4e1864"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:24:45.066931 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.066878 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:45.066931 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.066927 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l9x4\" (UniqueName: \"kubernetes.io/projected/23033390-d6eb-439a-a8dd-41925c4e1864-kube-api-access-4l9x4\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:45.067181 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.066942 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23033390-d6eb-439a-a8dd-41925c4e1864-util\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:45.752645 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.752609 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" event={"ID":"23033390-d6eb-439a-a8dd-41925c4e1864","Type":"ContainerDied","Data":"7b359c96b966f18528dc9a8717ab44dad6473e1e227697b9c3ef6c187aa1e0df"} Jul 31 14:24:45.752645 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.752630 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxn4k9" Jul 31 14:24:45.753173 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.752660 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b359c96b966f18528dc9a8717ab44dad6473e1e227697b9c3ef6c187aa1e0df" Jul 31 14:24:45.911630 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.911606 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:45.932640 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.932606 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:45.935945 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.935923 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:45.973004 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.972976 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqmdd\" (UniqueName: \"kubernetes.io/projected/5f21c2db-936e-455b-9750-61b10c90acd1-kube-api-access-qqmdd\") pod \"5f21c2db-936e-455b-9750-61b10c90acd1\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " Jul 31 14:24:45.973182 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973022 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-util\") pod \"5f21c2db-936e-455b-9750-61b10c90acd1\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " Jul 31 14:24:45.973256 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973201 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-bundle\") pod \"5f21c2db-936e-455b-9750-61b10c90acd1\" (UID: \"5f21c2db-936e-455b-9750-61b10c90acd1\") " Jul 31 14:24:45.973321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973260 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlg8q\" (UniqueName: \"kubernetes.io/projected/2978f7d9-f54f-48f1-9708-c528e4db68e1-kube-api-access-jlg8q\") pod \"2978f7d9-f54f-48f1-9708-c528e4db68e1\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " Jul 31 14:24:45.973321 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973296 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-bundle\") pod \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " Jul 31 14:24:45.973425 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973321 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-util\") pod \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " Jul 31 14:24:45.973425 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973363 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzmj5\" (UniqueName: \"kubernetes.io/projected/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-kube-api-access-dzmj5\") pod \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\" (UID: \"31b69f7c-d52e-48f4-8ad9-9c9d74b08528\") " Jul 31 14:24:45.973425 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973399 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-util\") pod \"2978f7d9-f54f-48f1-9708-c528e4db68e1\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " Jul 31 14:24:45.973574 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973448 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-bundle\") pod \"2978f7d9-f54f-48f1-9708-c528e4db68e1\" (UID: \"2978f7d9-f54f-48f1-9708-c528e4db68e1\") " Jul 31 14:24:45.973894 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973728 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-bundle" (OuterVolumeSpecName: "bundle") pod "5f21c2db-936e-455b-9750-61b10c90acd1" (UID: "5f21c2db-936e-455b-9750-61b10c90acd1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:24:45.974082 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.973885 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-bundle" (OuterVolumeSpecName: "bundle") pod "31b69f7c-d52e-48f4-8ad9-9c9d74b08528" (UID: "31b69f7c-d52e-48f4-8ad9-9c9d74b08528"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:24:45.974423 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.974388 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-bundle" (OuterVolumeSpecName: "bundle") pod "2978f7d9-f54f-48f1-9708-c528e4db68e1" (UID: "2978f7d9-f54f-48f1-9708-c528e4db68e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:24:45.976177 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.976134 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2978f7d9-f54f-48f1-9708-c528e4db68e1-kube-api-access-jlg8q" (OuterVolumeSpecName: "kube-api-access-jlg8q") pod "2978f7d9-f54f-48f1-9708-c528e4db68e1" (UID: "2978f7d9-f54f-48f1-9708-c528e4db68e1"). InnerVolumeSpecName "kube-api-access-jlg8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:24:45.976398 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.976373 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-kube-api-access-dzmj5" (OuterVolumeSpecName: "kube-api-access-dzmj5") pod "31b69f7c-d52e-48f4-8ad9-9c9d74b08528" (UID: "31b69f7c-d52e-48f4-8ad9-9c9d74b08528"). InnerVolumeSpecName "kube-api-access-dzmj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:24:45.976721 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.976695 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f21c2db-936e-455b-9750-61b10c90acd1-kube-api-access-qqmdd" (OuterVolumeSpecName: "kube-api-access-qqmdd") pod "5f21c2db-936e-455b-9750-61b10c90acd1" (UID: "5f21c2db-936e-455b-9750-61b10c90acd1"). InnerVolumeSpecName "kube-api-access-qqmdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:24:45.980957 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.980923 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-util" (OuterVolumeSpecName: "util") pod "31b69f7c-d52e-48f4-8ad9-9c9d74b08528" (UID: "31b69f7c-d52e-48f4-8ad9-9c9d74b08528"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:24:45.981521 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.981493 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-util" (OuterVolumeSpecName: "util") pod "5f21c2db-936e-455b-9750-61b10c90acd1" (UID: "5f21c2db-936e-455b-9750-61b10c90acd1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:24:45.983481 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:45.983428 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-util" (OuterVolumeSpecName: "util") pod "2978f7d9-f54f-48f1-9708-c528e4db68e1" (UID: "2978f7d9-f54f-48f1-9708-c528e4db68e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074419 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074462 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlg8q\" (UniqueName: \"kubernetes.io/projected/2978f7d9-f54f-48f1-9708-c528e4db68e1-kube-api-access-jlg8q\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074474 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074482 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-util\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074490 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzmj5\" (UniqueName: \"kubernetes.io/projected/31b69f7c-d52e-48f4-8ad9-9c9d74b08528-kube-api-access-dzmj5\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074498 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-util\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074506 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2978f7d9-f54f-48f1-9708-c528e4db68e1-bundle\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074514 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmdd\" (UniqueName: \"kubernetes.io/projected/5f21c2db-936e-455b-9750-61b10c90acd1-kube-api-access-qqmdd\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.074518 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.074522 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f21c2db-936e-455b-9750-61b10c90acd1-util\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:24:46.758456 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.758417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" event={"ID":"31b69f7c-d52e-48f4-8ad9-9c9d74b08528","Type":"ContainerDied","Data":"20aa3f537837709cd7e43f089fe167f66a1eb456e4c673e1a4314bed504b57a7"} Jul 31 14:24:46.758884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.758470 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20aa3f537837709cd7e43f089fe167f66a1eb456e4c673e1a4314bed504b57a7" Jul 31 14:24:46.758884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.758434 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbfg2t2" Jul 31 14:24:46.760371 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.760346 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" Jul 31 14:24:46.760464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.760386 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dv249" event={"ID":"5f21c2db-936e-455b-9750-61b10c90acd1","Type":"ContainerDied","Data":"b5638a4887d932086f14ec79a70c500b5215c959cf75c2a688f8b510c20d0981"} Jul 31 14:24:46.760464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.760420 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5638a4887d932086f14ec79a70c500b5215c959cf75c2a688f8b510c20d0981" Jul 31 14:24:46.762353 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.762331 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" event={"ID":"2978f7d9-f54f-48f1-9708-c528e4db68e1","Type":"ContainerDied","Data":"d09c272f6a1a383fc33b01ddd685e401e38c7c068848eaa67fc47beef152d70c"} Jul 31 14:24:46.762447 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.762374 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d09c272f6a1a383fc33b01ddd685e401e38c7c068848eaa67fc47beef152d70c" Jul 31 14:24:46.762447 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:46.762385 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053wjz67" Jul 31 14:24:48.321100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:48.321065 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:49.113470 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:49.113424 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:52.163526 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:52.163471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:52.384441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:52.384404 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:53.558201 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:53.558164 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:54.726188 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:54.726146 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:54.956218 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:54.956184 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:56.718346 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:56.718307 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:57.112740 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:57.112651 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:58.022837 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:58.022795 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:24:58.414947 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:24:58.414896 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:01.925177 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:01.925139 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:02.026359 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:02.026324 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:02.531148 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:02.531103 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:02.630961 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:02.630926 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:04.918684 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:04.918641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:04.971919 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:04.971865 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:08.257053 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257016 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27"] Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257364 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerName="pull" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257377 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerName="pull" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257388 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f21c2db-936e-455b-9750-61b10c90acd1" containerName="extract" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257393 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f21c2db-936e-455b-9750-61b10c90acd1" containerName="extract" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257400 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerName="util" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257405 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerName="util" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257412 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerName="util" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257417 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerName="util" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257425 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerName="extract" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257430 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerName="extract" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257450 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f21c2db-936e-455b-9750-61b10c90acd1" containerName="util" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257458 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f21c2db-936e-455b-9750-61b10c90acd1" containerName="util" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257468 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f21c2db-936e-455b-9750-61b10c90acd1" containerName="pull" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257477 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f21c2db-936e-455b-9750-61b10c90acd1" containerName="pull" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257498 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23033390-d6eb-439a-a8dd-41925c4e1864" containerName="extract" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257506 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23033390-d6eb-439a-a8dd-41925c4e1864" containerName="extract" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257518 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23033390-d6eb-439a-a8dd-41925c4e1864" containerName="util" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257526 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23033390-d6eb-439a-a8dd-41925c4e1864" containerName="util" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257539 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23033390-d6eb-439a-a8dd-41925c4e1864" containerName="pull" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257547 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23033390-d6eb-439a-a8dd-41925c4e1864" containerName="pull" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257557 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerName="pull" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257564 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerName="pull" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257573 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerName="extract" Jul 31 14:25:08.257586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257580 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerName="extract" Jul 31 14:25:08.258447 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257693 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f21c2db-936e-455b-9750-61b10c90acd1" containerName="extract" Jul 31 14:25:08.258447 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257707 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="31b69f7c-d52e-48f4-8ad9-9c9d74b08528" containerName="extract" Jul 31 14:25:08.258447 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257714 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23033390-d6eb-439a-a8dd-41925c4e1864" containerName="extract" Jul 31 14:25:08.258447 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.257722 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2978f7d9-f54f-48f1-9708-c528e4db68e1" containerName="extract" Jul 31 14:25:08.261816 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.261796 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.262684 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.262653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:08.264720 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.264701 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 31 14:25:08.264802 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.264705 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 31 14:25:08.264802 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.264784 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 31 14:25:08.266019 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.266002 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gx8vx\"" Jul 31 14:25:08.266105 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.266004 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 31 14:25:08.271030 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.271010 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27"] Jul 31 14:25:08.367446 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.367410 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rlh\" (UniqueName: \"kubernetes.io/projected/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-kube-api-access-r7rlh\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.367647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.367470 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.367647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.367553 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.415328 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.415288 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:08.468014 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.467978 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.468207 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.468027 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.468207 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.468058 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rlh\" (UniqueName: \"kubernetes.io/projected/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-kube-api-access-r7rlh\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.468673 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.468648 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.470591 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.470570 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.477990 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.477969 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rlh\" (UniqueName: \"kubernetes.io/projected/ea8d159c-6986-4c03-a7cc-568a99ccf0ce-kube-api-access-r7rlh\") pod \"kuadrant-console-plugin-598f86468b-wkc27\" (UID: \"ea8d159c-6986-4c03-a7cc-568a99ccf0ce\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.572162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.572059 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" Jul 31 14:25:08.707984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.707950 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27"] Jul 31 14:25:08.711466 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:25:08.711435 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea8d159c_6986_4c03_a7cc_568a99ccf0ce.slice/crio-670e3d072792da980f9eb1a47914cd57479c00430fd958e3b07c5a21350f9892 WatchSource:0}: Error finding container 670e3d072792da980f9eb1a47914cd57479c00430fd958e3b07c5a21350f9892: Status 404 returned error can't find the container with id 670e3d072792da980f9eb1a47914cd57479c00430fd958e3b07c5a21350f9892 Jul 31 14:25:08.867965 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:08.867935 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" event={"ID":"ea8d159c-6986-4c03-a7cc-568a99ccf0ce","Type":"ContainerStarted","Data":"670e3d072792da980f9eb1a47914cd57479c00430fd958e3b07c5a21350f9892"} Jul 31 14:25:09.139561 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:09.139470 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:09.260990 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:09.260953 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:13.858588 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:13.858491 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:13.899540 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:13.899498 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" event={"ID":"ea8d159c-6986-4c03-a7cc-568a99ccf0ce","Type":"ContainerStarted","Data":"ec8babf4ff43b366474fe1e436a67793c17f49795e97c8594dd990db5c989b11"} Jul 31 14:25:13.916058 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:13.916005 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wkc27" podStartSLOduration=1.081956439 podStartE2EDuration="5.915988567s" totalImagesPullingTime="4.834032128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:25:13.914800624 +0000 UTC m=+779.423179703" watchObservedRunningTime="2026-07-31 14:25:13.915988567 +0000 UTC m=+779.424367650" Jul 31 14:25:13.923833 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:13.923797 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:13.977956 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:13.977899 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:21.977067 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:21.977025 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:22.078689 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:22.078653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:22.575160 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:22.575125 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:22.633548 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:22.633515 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:25.575105 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:25.575070 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:25.675950 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:25.675871 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:28.577680 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:28.577651 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:28.678623 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:28.678592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:30.174293 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:30.174258 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:30.274446 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:30.274411 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:33.976017 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:33.975975 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:34.072339 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:34.072309 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:34.172201 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:34.172170 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:48.085509 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:48.085475 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:25:48.209137 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:25:48.209103 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:00.067104 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.067074 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-twvpm"] Jul 31 14:26:00.070040 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.069996 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:00.083517 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.083491 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-twvpm"] Jul 31 14:26:00.083721 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.083595 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:00.086698 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.086679 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 31 14:26:00.172034 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.171999 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-twvpm"] Jul 31 14:26:00.173349 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.173312 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:00.187313 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.187286 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcz9\" (UniqueName: \"kubernetes.io/projected/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-kube-api-access-pbcz9\") pod \"limitador-limitador-859c4d96d6-twvpm\" (UID: \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:00.187411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.187330 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-config-file\") pod \"limitador-limitador-859c4d96d6-twvpm\" (UID: \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:00.273619 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.273582 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:00.288798 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.288749 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcz9\" (UniqueName: \"kubernetes.io/projected/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-kube-api-access-pbcz9\") pod \"limitador-limitador-859c4d96d6-twvpm\" (UID: \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:00.288996 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.288809 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-config-file\") pod \"limitador-limitador-859c4d96d6-twvpm\" (UID: \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:00.289415 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.289397 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-config-file\") pod \"limitador-limitador-859c4d96d6-twvpm\" (UID: \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:00.297163 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.297144 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcz9\" (UniqueName: \"kubernetes.io/projected/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-kube-api-access-pbcz9\") pod \"limitador-limitador-859c4d96d6-twvpm\" (UID: \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:00.394162 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.394131 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:00.526331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.526226 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-twvpm"] Jul 31 14:26:00.528662 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:26:00.528638 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7c4d4a_47ee_494f_866c_a6cb5c8d388f.slice/crio-d8d5bb1dd21499439e153f893c295e7079af1c72d2bed9eb4d40636e7d67be51 WatchSource:0}: Error finding container d8d5bb1dd21499439e153f893c295e7079af1c72d2bed9eb4d40636e7d67be51: Status 404 returned error can't find the container with id d8d5bb1dd21499439e153f893c295e7079af1c72d2bed9eb4d40636e7d67be51 Jul 31 14:26:00.910326 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:00.910290 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:01.006078 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.006039 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:01.007411 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.007387 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ws48d"] Jul 31 14:26:01.011336 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.011317 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-ws48d" Jul 31 14:26:01.014090 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.014065 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wx7kw\"" Jul 31 14:26:01.017037 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.017014 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ws48d"] Jul 31 14:26:01.096658 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.096622 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27nd\" (UniqueName: \"kubernetes.io/projected/97a87968-7aad-43ad-9cf2-1432bb7fd796-kube-api-access-f27nd\") pod \"authorino-c548b5d7d-ws48d\" (UID: \"97a87968-7aad-43ad-9cf2-1432bb7fd796\") " pod="kuadrant-system/authorino-c548b5d7d-ws48d" Jul 31 14:26:01.122126 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.122092 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" event={"ID":"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f","Type":"ContainerStarted","Data":"d8d5bb1dd21499439e153f893c295e7079af1c72d2bed9eb4d40636e7d67be51"} Jul 31 14:26:01.197747 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.197658 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27nd\" (UniqueName: \"kubernetes.io/projected/97a87968-7aad-43ad-9cf2-1432bb7fd796-kube-api-access-f27nd\") pod \"authorino-c548b5d7d-ws48d\" (UID: \"97a87968-7aad-43ad-9cf2-1432bb7fd796\") " pod="kuadrant-system/authorino-c548b5d7d-ws48d" Jul 31 14:26:01.206838 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.206806 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27nd\" (UniqueName: \"kubernetes.io/projected/97a87968-7aad-43ad-9cf2-1432bb7fd796-kube-api-access-f27nd\") pod \"authorino-c548b5d7d-ws48d\" (UID: \"97a87968-7aad-43ad-9cf2-1432bb7fd796\") " pod="kuadrant-system/authorino-c548b5d7d-ws48d" Jul 31 14:26:01.322108 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.322071 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-ws48d" Jul 31 14:26:01.452242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:01.452216 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ws48d"] Jul 31 14:26:01.454395 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:26:01.454373 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97a87968_7aad_43ad_9cf2_1432bb7fd796.slice/crio-2ab8c2c9e741b3f6452dded6b84217433c46eb7dfcd0aadafd5f46c274ef503a WatchSource:0}: Error finding container 2ab8c2c9e741b3f6452dded6b84217433c46eb7dfcd0aadafd5f46c274ef503a: Status 404 returned error can't find the container with id 2ab8c2c9e741b3f6452dded6b84217433c46eb7dfcd0aadafd5f46c274ef503a Jul 31 14:26:02.128712 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:02.128678 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-ws48d" event={"ID":"97a87968-7aad-43ad-9cf2-1432bb7fd796","Type":"ContainerStarted","Data":"2ab8c2c9e741b3f6452dded6b84217433c46eb7dfcd0aadafd5f46c274ef503a"} Jul 31 14:26:04.140730 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:04.140691 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" event={"ID":"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f","Type":"ContainerStarted","Data":"eb0492170d76f25e28009655ab3ddc0670d492b8e63d0a03c56f164166af5624"} Jul 31 14:26:04.141121 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:04.140849 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:04.157777 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:04.157724 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" podStartSLOduration=1.219264309 podStartE2EDuration="4.157707754s" totalImagesPullingTime="2.938443445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:26:04.156342705 +0000 UTC m=+829.664721770" watchObservedRunningTime="2026-07-31 14:26:04.157707754 +0000 UTC m=+829.666086819" Jul 31 14:26:06.151846 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:06.151809 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-ws48d" event={"ID":"97a87968-7aad-43ad-9cf2-1432bb7fd796","Type":"ContainerStarted","Data":"c9366acc60ded99c15ddafd7b4f7728d275d383927bf2dc23b390fe178a104d5"} Jul 31 14:26:06.168093 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:06.168046 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-ws48d" podStartSLOduration=2.084862914 podStartE2EDuration="6.168032466s" totalImagesPullingTime="4.083169552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:26:06.166401346 +0000 UTC m=+831.674780411" watchObservedRunningTime="2026-07-31 14:26:06.168032466 +0000 UTC m=+831.676411531" Jul 31 14:26:15.147460 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:15.147428 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:16.474296 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:16.474261 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-twvpm"] Jul 31 14:26:16.474697 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:16.474457 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" podUID="2a7c4d4a-47ee-494f-866c-a6cb5c8d388f" containerName="limitador" containerID="cri-o://eb0492170d76f25e28009655ab3ddc0670d492b8e63d0a03c56f164166af5624" gracePeriod=30 Jul 31 14:26:17.207294 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.207263 2558 generic.go:356] "Generic (PLEG): container finished" podID="2a7c4d4a-47ee-494f-866c-a6cb5c8d388f" containerID="eb0492170d76f25e28009655ab3ddc0670d492b8e63d0a03c56f164166af5624" exitCode=0 Jul 31 14:26:17.207464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.207337 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" event={"ID":"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f","Type":"ContainerDied","Data":"eb0492170d76f25e28009655ab3ddc0670d492b8e63d0a03c56f164166af5624"} Jul 31 14:26:17.426193 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.426170 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:17.547416 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.547323 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbcz9\" (UniqueName: \"kubernetes.io/projected/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-kube-api-access-pbcz9\") pod \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\" (UID: \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\") " Jul 31 14:26:17.547416 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.547392 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-config-file\") pod \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\" (UID: \"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f\") " Jul 31 14:26:17.547860 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.547675 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-config-file" (OuterVolumeSpecName: "config-file") pod "2a7c4d4a-47ee-494f-866c-a6cb5c8d388f" (UID: "2a7c4d4a-47ee-494f-866c-a6cb5c8d388f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:26:17.549483 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.549459 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-kube-api-access-pbcz9" (OuterVolumeSpecName: "kube-api-access-pbcz9") pod "2a7c4d4a-47ee-494f-866c-a6cb5c8d388f" (UID: "2a7c4d4a-47ee-494f-866c-a6cb5c8d388f"). InnerVolumeSpecName "kube-api-access-pbcz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:26:17.649032 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.648997 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbcz9\" (UniqueName: \"kubernetes.io/projected/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-kube-api-access-pbcz9\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:26:17.649032 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:17.649026 2558 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f-config-file\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:26:18.213404 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:18.213373 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" Jul 31 14:26:18.213577 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:18.213401 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-twvpm" event={"ID":"2a7c4d4a-47ee-494f-866c-a6cb5c8d388f","Type":"ContainerDied","Data":"d8d5bb1dd21499439e153f893c295e7079af1c72d2bed9eb4d40636e7d67be51"} Jul 31 14:26:18.213577 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:18.213461 2558 scope.go:117] "RemoveContainer" containerID="eb0492170d76f25e28009655ab3ddc0670d492b8e63d0a03c56f164166af5624" Jul 31 14:26:18.233883 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:18.233859 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-twvpm"] Jul 31 14:26:18.236862 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:18.236837 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-twvpm"] Jul 31 14:26:18.983236 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:18.983200 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7c4d4a-47ee-494f-866c-a6cb5c8d388f" path="/var/lib/kubelet/pods/2a7c4d4a-47ee-494f-866c-a6cb5c8d388f/volumes" Jul 31 14:26:21.274159 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:21.274123 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:21.404249 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:21.404212 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:21.754945 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:21.754895 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:26:51.469244 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:26:51.469200 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:31.644465 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:31.644428 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:31.883709 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:31.883672 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-jznxl"] Jul 31 14:27:31.884063 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:31.884046 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7c4d4a-47ee-494f-866c-a6cb5c8d388f" containerName="limitador" Jul 31 14:27:31.884063 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:31.884062 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7c4d4a-47ee-494f-866c-a6cb5c8d388f" containerName="limitador" Jul 31 14:27:31.884223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:31.884126 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7c4d4a-47ee-494f-866c-a6cb5c8d388f" containerName="limitador" Jul 31 14:27:31.886975 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:31.886956 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" Jul 31 14:27:31.893513 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:31.893485 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-jznxl"] Jul 31 14:27:31.959152 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:31.959054 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tshp\" (UniqueName: \"kubernetes.io/projected/981b7bf9-e1ff-425b-bd6b-666d9d110081-kube-api-access-2tshp\") pod \"authorino-5dc7ccb5f-jznxl\" (UID: \"981b7bf9-e1ff-425b-bd6b-666d9d110081\") " pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" Jul 31 14:27:32.060045 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.060009 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tshp\" (UniqueName: \"kubernetes.io/projected/981b7bf9-e1ff-425b-bd6b-666d9d110081-kube-api-access-2tshp\") pod \"authorino-5dc7ccb5f-jznxl\" (UID: \"981b7bf9-e1ff-425b-bd6b-666d9d110081\") " pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" Jul 31 14:27:32.068030 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.068007 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tshp\" (UniqueName: \"kubernetes.io/projected/981b7bf9-e1ff-425b-bd6b-666d9d110081-kube-api-access-2tshp\") pod \"authorino-5dc7ccb5f-jznxl\" (UID: \"981b7bf9-e1ff-425b-bd6b-666d9d110081\") " pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" Jul 31 14:27:32.121201 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.121173 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-jznxl"] Jul 31 14:27:32.121442 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.121431 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" Jul 31 14:27:32.147930 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.146931 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-649b74d77d-6rxwj"] Jul 31 14:27:32.154611 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.154560 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-649b74d77d-6rxwj" Jul 31 14:27:32.156138 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.156111 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-649b74d77d-6rxwj"] Jul 31 14:27:32.195893 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.195824 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-649b74d77d-6rxwj"] Jul 31 14:27:32.196142 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:27:32.196121 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-t6ph6], unattached volumes=[], failed to process volumes=[kube-api-access-t6ph6]: context canceled" pod="kuadrant-system/authorino-649b74d77d-6rxwj" podUID="4f4172b7-886e-4e2b-8be6-ebb092e979f6" Jul 31 14:27:32.258602 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.258577 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-jznxl"] Jul 31 14:27:32.260500 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:27:32.260468 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod981b7bf9_e1ff_425b_bd6b_666d9d110081.slice/crio-79db6180e3ba86a1b0171453bd7484f9525468aba480c47988799867c8d13659 WatchSource:0}: Error finding container 79db6180e3ba86a1b0171453bd7484f9525468aba480c47988799867c8d13659: Status 404 returned error can't find the container with id 79db6180e3ba86a1b0171453bd7484f9525468aba480c47988799867c8d13659 Jul 31 14:27:32.261457 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.261427 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ph6\" (UniqueName: \"kubernetes.io/projected/4f4172b7-886e-4e2b-8be6-ebb092e979f6-kube-api-access-t6ph6\") pod \"authorino-649b74d77d-6rxwj\" (UID: \"4f4172b7-886e-4e2b-8be6-ebb092e979f6\") " pod="kuadrant-system/authorino-649b74d77d-6rxwj" Jul 31 14:27:32.362443 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.362409 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ph6\" (UniqueName: \"kubernetes.io/projected/4f4172b7-886e-4e2b-8be6-ebb092e979f6-kube-api-access-t6ph6\") pod \"authorino-649b74d77d-6rxwj\" (UID: \"4f4172b7-886e-4e2b-8be6-ebb092e979f6\") " pod="kuadrant-system/authorino-649b74d77d-6rxwj" Jul 31 14:27:32.371267 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.371230 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ph6\" (UniqueName: \"kubernetes.io/projected/4f4172b7-886e-4e2b-8be6-ebb092e979f6-kube-api-access-t6ph6\") pod \"authorino-649b74d77d-6rxwj\" (UID: \"4f4172b7-886e-4e2b-8be6-ebb092e979f6\") " pod="kuadrant-system/authorino-649b74d77d-6rxwj" Jul 31 14:27:32.573236 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.573138 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" event={"ID":"981b7bf9-e1ff-425b-bd6b-666d9d110081","Type":"ContainerStarted","Data":"79db6180e3ba86a1b0171453bd7484f9525468aba480c47988799867c8d13659"} Jul 31 14:27:32.573236 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.573180 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-649b74d77d-6rxwj" Jul 31 14:27:32.580324 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.580305 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-649b74d77d-6rxwj" Jul 31 14:27:32.665170 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.665138 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6ph6\" (UniqueName: \"kubernetes.io/projected/4f4172b7-886e-4e2b-8be6-ebb092e979f6-kube-api-access-t6ph6\") pod \"4f4172b7-886e-4e2b-8be6-ebb092e979f6\" (UID: \"4f4172b7-886e-4e2b-8be6-ebb092e979f6\") " Jul 31 14:27:32.667158 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.667135 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4172b7-886e-4e2b-8be6-ebb092e979f6-kube-api-access-t6ph6" (OuterVolumeSpecName: "kube-api-access-t6ph6") pod "4f4172b7-886e-4e2b-8be6-ebb092e979f6" (UID: "4f4172b7-886e-4e2b-8be6-ebb092e979f6"). InnerVolumeSpecName "kube-api-access-t6ph6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:27:32.765742 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.765707 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ph6\" (UniqueName: \"kubernetes.io/projected/4f4172b7-886e-4e2b-8be6-ebb092e979f6-kube-api-access-t6ph6\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:27:32.976837 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:32.976792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:33.577351 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:33.577313 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-649b74d77d-6rxwj" Jul 31 14:27:33.605108 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:33.605076 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-649b74d77d-6rxwj"] Jul 31 14:27:33.608037 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:33.608011 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-649b74d77d-6rxwj"] Jul 31 14:27:34.377887 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.377851 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ws48d"] Jul 31 14:27:34.378273 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.378042 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-ws48d" podUID="97a87968-7aad-43ad-9cf2-1432bb7fd796" containerName="authorino" containerID="cri-o://c9366acc60ded99c15ddafd7b4f7728d275d383927bf2dc23b390fe178a104d5" gracePeriod=30 Jul 31 14:27:34.431299 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.431263 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:34.584648 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.584610 2558 generic.go:356] "Generic (PLEG): container finished" podID="97a87968-7aad-43ad-9cf2-1432bb7fd796" containerID="c9366acc60ded99c15ddafd7b4f7728d275d383927bf2dc23b390fe178a104d5" exitCode=0 Jul 31 14:27:34.584894 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.584679 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-ws48d" event={"ID":"97a87968-7aad-43ad-9cf2-1432bb7fd796","Type":"ContainerDied","Data":"c9366acc60ded99c15ddafd7b4f7728d275d383927bf2dc23b390fe178a104d5"} Jul 31 14:27:34.628558 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.628503 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-ws48d" Jul 31 14:27:34.682501 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.682471 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f27nd\" (UniqueName: \"kubernetes.io/projected/97a87968-7aad-43ad-9cf2-1432bb7fd796-kube-api-access-f27nd\") pod \"97a87968-7aad-43ad-9cf2-1432bb7fd796\" (UID: \"97a87968-7aad-43ad-9cf2-1432bb7fd796\") " Jul 31 14:27:34.684405 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.684384 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a87968-7aad-43ad-9cf2-1432bb7fd796-kube-api-access-f27nd" (OuterVolumeSpecName: "kube-api-access-f27nd") pod "97a87968-7aad-43ad-9cf2-1432bb7fd796" (UID: "97a87968-7aad-43ad-9cf2-1432bb7fd796"). InnerVolumeSpecName "kube-api-access-f27nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:27:34.783215 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.783172 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f27nd\" (UniqueName: \"kubernetes.io/projected/97a87968-7aad-43ad-9cf2-1432bb7fd796-kube-api-access-f27nd\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:27:34.982749 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:34.982670 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f4172b7-886e-4e2b-8be6-ebb092e979f6" path="/var/lib/kubelet/pods/4f4172b7-886e-4e2b-8be6-ebb092e979f6/volumes" Jul 31 14:27:35.590576 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:35.590541 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-ws48d" event={"ID":"97a87968-7aad-43ad-9cf2-1432bb7fd796","Type":"ContainerDied","Data":"2ab8c2c9e741b3f6452dded6b84217433c46eb7dfcd0aadafd5f46c274ef503a"} Jul 31 14:27:35.590576 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:35.590575 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-ws48d" Jul 31 14:27:35.591070 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:35.590595 2558 scope.go:117] "RemoveContainer" containerID="c9366acc60ded99c15ddafd7b4f7728d275d383927bf2dc23b390fe178a104d5" Jul 31 14:27:35.607090 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:35.607067 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ws48d"] Jul 31 14:27:35.610320 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:35.610298 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ws48d"] Jul 31 14:27:36.987072 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:36.983707 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a87968-7aad-43ad-9cf2-1432bb7fd796" path="/var/lib/kubelet/pods/97a87968-7aad-43ad-9cf2-1432bb7fd796/volumes" Jul 31 14:27:39.209025 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.208986 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:39.226081 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.226047 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8654599dc6-mq96d"] Jul 31 14:27:39.226455 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.226437 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a87968-7aad-43ad-9cf2-1432bb7fd796" containerName="authorino" Jul 31 14:27:39.226455 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.226454 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a87968-7aad-43ad-9cf2-1432bb7fd796" containerName="authorino" Jul 31 14:27:39.226566 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.226517 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a87968-7aad-43ad-9cf2-1432bb7fd796" containerName="authorino" Jul 31 14:27:39.230752 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.230735 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:39.233533 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.233505 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 14:27:39.233646 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.233557 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 14:27:39.233646 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.233561 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rvdts\"" Jul 31 14:27:39.239369 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.239348 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8654599dc6-mq96d"] Jul 31 14:27:39.308894 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.308857 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:39.319013 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.318988 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hddt\" (UniqueName: \"kubernetes.io/projected/d100af02-6609-4a96-b03e-102862820abe-kube-api-access-5hddt\") pod \"maas-controller-8654599dc6-mq96d\" (UID: \"d100af02-6609-4a96-b03e-102862820abe\") " pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:39.319122 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.319059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d100af02-6609-4a96-b03e-102862820abe-webhook-cert\") pod \"maas-controller-8654599dc6-mq96d\" (UID: \"d100af02-6609-4a96-b03e-102862820abe\") " pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:39.363689 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.363653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:39.406021 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.405988 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f846468f4-nqqzl"] Jul 31 14:27:39.409219 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.409191 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:39.419516 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.419487 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d100af02-6609-4a96-b03e-102862820abe-webhook-cert\") pod \"maas-controller-8654599dc6-mq96d\" (UID: \"d100af02-6609-4a96-b03e-102862820abe\") " pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:39.419655 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.419567 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hddt\" (UniqueName: \"kubernetes.io/projected/d100af02-6609-4a96-b03e-102862820abe-kube-api-access-5hddt\") pod \"maas-controller-8654599dc6-mq96d\" (UID: \"d100af02-6609-4a96-b03e-102862820abe\") " pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:39.420586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.420554 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f846468f4-nqqzl"] Jul 31 14:27:39.422400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.422377 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d100af02-6609-4a96-b03e-102862820abe-webhook-cert\") pod \"maas-controller-8654599dc6-mq96d\" (UID: \"d100af02-6609-4a96-b03e-102862820abe\") " pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:39.430234 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.430203 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:39.431063 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.431046 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hddt\" (UniqueName: \"kubernetes.io/projected/d100af02-6609-4a96-b03e-102862820abe-kube-api-access-5hddt\") pod \"maas-controller-8654599dc6-mq96d\" (UID: \"d100af02-6609-4a96-b03e-102862820abe\") " pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:39.520362 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.520272 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpmr\" (UniqueName: \"kubernetes.io/projected/71733483-ded1-487a-9106-aa92b9fe849b-kube-api-access-rqpmr\") pod \"maas-controller-f846468f4-nqqzl\" (UID: \"71733483-ded1-487a-9106-aa92b9fe849b\") " pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:39.520528 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.520389 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71733483-ded1-487a-9106-aa92b9fe849b-webhook-cert\") pod \"maas-controller-f846468f4-nqqzl\" (UID: \"71733483-ded1-487a-9106-aa92b9fe849b\") " pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:39.542272 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.542244 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:39.621641 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.621616 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71733483-ded1-487a-9106-aa92b9fe849b-webhook-cert\") pod \"maas-controller-f846468f4-nqqzl\" (UID: \"71733483-ded1-487a-9106-aa92b9fe849b\") " pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:39.621794 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.621652 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpmr\" (UniqueName: \"kubernetes.io/projected/71733483-ded1-487a-9106-aa92b9fe849b-kube-api-access-rqpmr\") pod \"maas-controller-f846468f4-nqqzl\" (UID: \"71733483-ded1-487a-9106-aa92b9fe849b\") " pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:39.624627 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.624574 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71733483-ded1-487a-9106-aa92b9fe849b-webhook-cert\") pod \"maas-controller-f846468f4-nqqzl\" (UID: \"71733483-ded1-487a-9106-aa92b9fe849b\") " pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:39.630315 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.630290 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpmr\" (UniqueName: \"kubernetes.io/projected/71733483-ded1-487a-9106-aa92b9fe849b-kube-api-access-rqpmr\") pod \"maas-controller-f846468f4-nqqzl\" (UID: \"71733483-ded1-487a-9106-aa92b9fe849b\") " pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:39.670830 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.670800 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8654599dc6-mq96d"] Jul 31 14:27:39.672308 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:27:39.672285 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd100af02_6609_4a96_b03e_102862820abe.slice/crio-d35a86bb57badf2c61dd162c949ff240f11cf6e70bc36a01601bf1049dc63dac WatchSource:0}: Error finding container d35a86bb57badf2c61dd162c949ff240f11cf6e70bc36a01601bf1049dc63dac: Status 404 returned error can't find the container with id d35a86bb57badf2c61dd162c949ff240f11cf6e70bc36a01601bf1049dc63dac Jul 31 14:27:39.719660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.719634 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:39.844694 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:39.844662 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f846468f4-nqqzl"] Jul 31 14:27:39.845866 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:27:39.845834 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71733483_ded1_487a_9106_aa92b9fe849b.slice/crio-4009234066a0cf8a4781fb53bf888da1b726e92872223bec94db02c6b810204f WatchSource:0}: Error finding container 4009234066a0cf8a4781fb53bf888da1b726e92872223bec94db02c6b810204f: Status 404 returned error can't find the container with id 4009234066a0cf8a4781fb53bf888da1b726e92872223bec94db02c6b810204f Jul 31 14:27:40.622895 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:40.622214 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-nqqzl" event={"ID":"71733483-ded1-487a-9106-aa92b9fe849b","Type":"ContainerStarted","Data":"4009234066a0cf8a4781fb53bf888da1b726e92872223bec94db02c6b810204f"} Jul 31 14:27:40.624857 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:40.624799 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8654599dc6-mq96d" event={"ID":"d100af02-6609-4a96-b03e-102862820abe","Type":"ContainerStarted","Data":"d35a86bb57badf2c61dd162c949ff240f11cf6e70bc36a01601bf1049dc63dac"} Jul 31 14:27:42.376842 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:42.376805 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:42.466862 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:42.466775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:42.635854 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:42.635816 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8654599dc6-mq96d" event={"ID":"d100af02-6609-4a96-b03e-102862820abe","Type":"ContainerStarted","Data":"31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8"} Jul 31 14:27:42.636066 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:42.635887 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:42.637493 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:42.637464 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-nqqzl" event={"ID":"71733483-ded1-487a-9106-aa92b9fe849b","Type":"ContainerStarted","Data":"4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5"} Jul 31 14:27:42.637623 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:42.637573 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:42.658519 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:42.658346 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8654599dc6-mq96d" podStartSLOduration=1.124859299 podStartE2EDuration="3.658322364s" totalImagesPullingTime="2.533463065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:27:42.651754541 +0000 UTC m=+928.160133608" watchObservedRunningTime="2026-07-31 14:27:42.658322364 +0000 UTC m=+928.166701431" Jul 31 14:27:42.668788 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:42.668736 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f846468f4-nqqzl" podStartSLOduration=1.3162215640000001 podStartE2EDuration="3.668723538s" totalImagesPullingTime="2.352501974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:27:42.668621109 +0000 UTC m=+928.177000174" watchObservedRunningTime="2026-07-31 14:27:42.668723538 +0000 UTC m=+928.177102603" Jul 31 14:27:49.468801 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.468768 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ddb88b868-6c9h7"] Jul 31 14:27:49.472426 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.472403 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.474992 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.474970 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 31 14:27:49.476359 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.476340 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bv27n\"" Jul 31 14:27:49.476476 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.476371 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 31 14:27:49.476476 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.476464 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 31 14:27:49.476575 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.476537 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 31 14:27:49.476653 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.476632 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 31 14:27:49.476777 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.476640 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 31 14:27:49.476777 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.476719 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 31 14:27:49.481650 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.481630 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 31 14:27:49.483936 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.483918 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ddb88b868-6c9h7"] Jul 31 14:27:49.515601 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.515570 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:49.572246 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.572207 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:49.609408 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.609370 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9661f8-768f-44b3-917f-607859268da5-console-serving-cert\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.609525 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.609413 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-oauth-serving-cert\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.609525 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.609447 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n22v\" (UniqueName: \"kubernetes.io/projected/2b9661f8-768f-44b3-917f-607859268da5-kube-api-access-7n22v\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.609525 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.609488 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-service-ca\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.609525 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.609506 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-trusted-ca-bundle\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.609525 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.609525 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-console-config\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.609708 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.609547 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b9661f8-768f-44b3-917f-607859268da5-console-oauth-config\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.667784 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.667753 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:49.710244 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.710205 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9661f8-768f-44b3-917f-607859268da5-console-serving-cert\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.710244 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.710246 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-oauth-serving-cert\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.710495 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.710290 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n22v\" (UniqueName: \"kubernetes.io/projected/2b9661f8-768f-44b3-917f-607859268da5-kube-api-access-7n22v\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.710495 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.710332 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-service-ca\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.710495 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.710358 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-trusted-ca-bundle\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.710495 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.710388 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-console-config\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.710495 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.710419 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b9661f8-768f-44b3-917f-607859268da5-console-oauth-config\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.711183 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.711154 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-oauth-serving-cert\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.711341 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.711314 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-service-ca\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.711450 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.711376 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-trusted-ca-bundle\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.711802 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.711779 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b9661f8-768f-44b3-917f-607859268da5-console-config\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.712844 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.712823 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9661f8-768f-44b3-917f-607859268da5-console-serving-cert\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.712934 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.712876 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b9661f8-768f-44b3-917f-607859268da5-console-oauth-config\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.719770 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.719704 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n22v\" (UniqueName: \"kubernetes.io/projected/2b9661f8-768f-44b3-917f-607859268da5-kube-api-access-7n22v\") pod \"console-ddb88b868-6c9h7\" (UID: \"2b9661f8-768f-44b3-917f-607859268da5\") " pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.720340 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.720315 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:49.783550 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.783515 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:49.915502 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:49.915475 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ddb88b868-6c9h7"] Jul 31 14:27:49.917879 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:27:49.917854 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9661f8_768f_44b3_917f_607859268da5.slice/crio-d5582d9e8aa04e9f97394591c1c55e525b051826a53f1cdd0d7cf30e7c99f290 WatchSource:0}: Error finding container d5582d9e8aa04e9f97394591c1c55e525b051826a53f1cdd0d7cf30e7c99f290: Status 404 returned error can't find the container with id d5582d9e8aa04e9f97394591c1c55e525b051826a53f1cdd0d7cf30e7c99f290 Jul 31 14:27:50.272692 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.272573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:50.329019 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.328987 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:50.679316 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.679278 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddb88b868-6c9h7" event={"ID":"2b9661f8-768f-44b3-917f-607859268da5","Type":"ContainerStarted","Data":"ae0b0246690c353db337e02c3eb14e01a488ad455e2e901ec9429c8a422d1e00"} Jul 31 14:27:50.679746 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.679337 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddb88b868-6c9h7" event={"ID":"2b9661f8-768f-44b3-917f-607859268da5","Type":"ContainerStarted","Data":"d5582d9e8aa04e9f97394591c1c55e525b051826a53f1cdd0d7cf30e7c99f290"} Jul 31 14:27:50.681457 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.681399 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" event={"ID":"981b7bf9-e1ff-425b-bd6b-666d9d110081","Type":"ContainerStarted","Data":"bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649"} Jul 31 14:27:50.681457 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.681443 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" podUID="981b7bf9-e1ff-425b-bd6b-666d9d110081" containerName="authorino" containerID="cri-o://bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649" gracePeriod=30 Jul 31 14:27:50.700646 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.700601 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ddb88b868-6c9h7" podStartSLOduration=1.700586653 podStartE2EDuration="1.700586653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:27:50.698009713 +0000 UTC m=+936.206388779" watchObservedRunningTime="2026-07-31 14:27:50.700586653 +0000 UTC m=+936.208965718" Jul 31 14:27:50.713794 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.713750 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" podStartSLOduration=1.397136762 podStartE2EDuration="19.713735857s" totalImagesPullingTime="18.316599095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:27:50.711865387 +0000 UTC m=+936.220244452" watchObservedRunningTime="2026-07-31 14:27:50.713735857 +0000 UTC m=+936.222114963" Jul 31 14:27:50.930061 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:50.930037 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" Jul 31 14:27:51.021400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.021369 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tshp\" (UniqueName: \"kubernetes.io/projected/981b7bf9-e1ff-425b-bd6b-666d9d110081-kube-api-access-2tshp\") pod \"981b7bf9-e1ff-425b-bd6b-666d9d110081\" (UID: \"981b7bf9-e1ff-425b-bd6b-666d9d110081\") " Jul 31 14:27:51.023420 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.023393 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981b7bf9-e1ff-425b-bd6b-666d9d110081-kube-api-access-2tshp" (OuterVolumeSpecName: "kube-api-access-2tshp") pod "981b7bf9-e1ff-425b-bd6b-666d9d110081" (UID: "981b7bf9-e1ff-425b-bd6b-666d9d110081"). InnerVolumeSpecName "kube-api-access-2tshp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:27:51.122878 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.122846 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tshp\" (UniqueName: \"kubernetes.io/projected/981b7bf9-e1ff-425b-bd6b-666d9d110081-kube-api-access-2tshp\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:27:51.687210 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.687168 2558 generic.go:356] "Generic (PLEG): container finished" podID="981b7bf9-e1ff-425b-bd6b-666d9d110081" containerID="bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649" exitCode=0 Jul 31 14:27:51.687609 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.687218 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" Jul 31 14:27:51.687609 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.687252 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" event={"ID":"981b7bf9-e1ff-425b-bd6b-666d9d110081","Type":"ContainerDied","Data":"bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649"} Jul 31 14:27:51.687609 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.687300 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-jznxl" event={"ID":"981b7bf9-e1ff-425b-bd6b-666d9d110081","Type":"ContainerDied","Data":"79db6180e3ba86a1b0171453bd7484f9525468aba480c47988799867c8d13659"} Jul 31 14:27:51.687609 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.687322 2558 scope.go:117] "RemoveContainer" containerID="bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649" Jul 31 14:27:51.696470 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.696452 2558 scope.go:117] "RemoveContainer" containerID="bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649" Jul 31 14:27:51.696745 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:27:51.696726 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649\": container with ID starting with bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649 not found: ID does not exist" containerID="bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649" Jul 31 14:27:51.696799 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.696753 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649"} err="failed to get container status \"bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649\": rpc error: code = NotFound desc = could not find container \"bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649\": container with ID starting with bb0f39aa71076f874b906483eff55bf9e44bd41ad9d8c471d525238d99d8e649 not found: ID does not exist" Jul 31 14:27:51.708252 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.708230 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-jznxl"] Jul 31 14:27:51.713577 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:51.713555 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-jznxl"] Jul 31 14:27:52.982691 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:52.982652 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="981b7bf9-e1ff-425b-bd6b-666d9d110081" path="/var/lib/kubelet/pods/981b7bf9-e1ff-425b-bd6b-666d9d110081/volumes" Jul 31 14:27:53.381490 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:53.381450 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:53.483812 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:53.483778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:27:53.644852 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:53.644770 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:53.644852 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:53.644843 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:27:53.703316 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:53.703280 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8654599dc6-mq96d"] Jul 31 14:27:53.703561 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:53.703518 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8654599dc6-mq96d" podUID="d100af02-6609-4a96-b03e-102862820abe" containerName="manager" containerID="cri-o://31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8" gracePeriod=30 Jul 31 14:27:53.950556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:53.950526 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:54.045937 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.045884 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hddt\" (UniqueName: \"kubernetes.io/projected/d100af02-6609-4a96-b03e-102862820abe-kube-api-access-5hddt\") pod \"d100af02-6609-4a96-b03e-102862820abe\" (UID: \"d100af02-6609-4a96-b03e-102862820abe\") " Jul 31 14:27:54.046298 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.045985 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d100af02-6609-4a96-b03e-102862820abe-webhook-cert\") pod \"d100af02-6609-4a96-b03e-102862820abe\" (UID: \"d100af02-6609-4a96-b03e-102862820abe\") " Jul 31 14:27:54.048119 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.048090 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d100af02-6609-4a96-b03e-102862820abe-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d100af02-6609-4a96-b03e-102862820abe" (UID: "d100af02-6609-4a96-b03e-102862820abe"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:27:54.048207 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.048188 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d100af02-6609-4a96-b03e-102862820abe-kube-api-access-5hddt" (OuterVolumeSpecName: "kube-api-access-5hddt") pod "d100af02-6609-4a96-b03e-102862820abe" (UID: "d100af02-6609-4a96-b03e-102862820abe"). InnerVolumeSpecName "kube-api-access-5hddt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:27:54.146655 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.146623 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hddt\" (UniqueName: \"kubernetes.io/projected/d100af02-6609-4a96-b03e-102862820abe-kube-api-access-5hddt\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:27:54.146655 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.146649 2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d100af02-6609-4a96-b03e-102862820abe-webhook-cert\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:27:54.704658 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.704622 2558 generic.go:356] "Generic (PLEG): container finished" podID="d100af02-6609-4a96-b03e-102862820abe" containerID="31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8" exitCode=0 Jul 31 14:27:54.704845 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.704670 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8654599dc6-mq96d" Jul 31 14:27:54.704845 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.704703 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8654599dc6-mq96d" event={"ID":"d100af02-6609-4a96-b03e-102862820abe","Type":"ContainerDied","Data":"31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8"} Jul 31 14:27:54.704845 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.704759 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8654599dc6-mq96d" event={"ID":"d100af02-6609-4a96-b03e-102862820abe","Type":"ContainerDied","Data":"d35a86bb57badf2c61dd162c949ff240f11cf6e70bc36a01601bf1049dc63dac"} Jul 31 14:27:54.704845 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.704783 2558 scope.go:117] "RemoveContainer" containerID="31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8" Jul 31 14:27:54.714354 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.714327 2558 scope.go:117] "RemoveContainer" containerID="31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8" Jul 31 14:27:54.714691 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:27:54.714674 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8\": container with ID starting with 31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8 not found: ID does not exist" containerID="31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8" Jul 31 14:27:54.714744 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.714702 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8"} err="failed to get container status \"31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8\": rpc error: code = NotFound desc = could not find container \"31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8\": container with ID starting with 31f5771525e8e3ce48b73ca07959fa0ee1fc9cde3ec3bf1154efcf0e0fb5e8d8 not found: ID does not exist" Jul 31 14:27:54.726205 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.726178 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8654599dc6-mq96d"] Jul 31 14:27:54.729452 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.729417 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8654599dc6-mq96d"] Jul 31 14:27:54.983183 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:54.983109 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d100af02-6609-4a96-b03e-102862820abe" path="/var/lib/kubelet/pods/d100af02-6609-4a96-b03e-102862820abe/volumes" Jul 31 14:27:59.784657 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:59.784621 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:59.785102 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:59.784697 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:27:59.789519 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:27:59.789500 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:28:00.744835 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:00.744798 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ddb88b868-6c9h7" Jul 31 14:28:00.889381 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:00.889352 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:00.941170 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:00.941134 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:00.997422 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:00.997328 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:15.047319 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:15.047281 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:15.146252 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:15.146219 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:16.654533 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.654493 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:16.720537 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.720503 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh"] Jul 31 14:28:16.720929 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.720896 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d100af02-6609-4a96-b03e-102862820abe" containerName="manager" Jul 31 14:28:16.720929 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.720928 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d100af02-6609-4a96-b03e-102862820abe" containerName="manager" Jul 31 14:28:16.721097 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.720946 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981b7bf9-e1ff-425b-bd6b-666d9d110081" containerName="authorino" Jul 31 14:28:16.721097 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.720954 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="981b7bf9-e1ff-425b-bd6b-666d9d110081" containerName="authorino" Jul 31 14:28:16.721097 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.721038 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d100af02-6609-4a96-b03e-102862820abe" containerName="manager" Jul 31 14:28:16.721097 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.721048 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="981b7bf9-e1ff-425b-bd6b-666d9d110081" containerName="authorino" Jul 31 14:28:16.727380 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.727362 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:16.730500 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.730475 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 31 14:28:16.730500 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.730493 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cl2ws\"" Jul 31 14:28:16.730664 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.730482 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 31 14:28:16.731536 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.731517 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 31 14:28:16.736267 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.736246 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh"] Jul 31 14:28:16.837516 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.837481 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20df45c3-27b5-4510-883c-3e9354e85fbe-openshift-service-ca\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:16.837728 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.837537 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20df45c3-27b5-4510-883c-3e9354e85fbe-maas-api-tls\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:16.837728 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.837612 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7s6f\" (UniqueName: \"kubernetes.io/projected/20df45c3-27b5-4510-883c-3e9354e85fbe-kube-api-access-k7s6f\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:16.938499 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.938414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20df45c3-27b5-4510-883c-3e9354e85fbe-maas-api-tls\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:16.938499 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.938461 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7s6f\" (UniqueName: \"kubernetes.io/projected/20df45c3-27b5-4510-883c-3e9354e85fbe-kube-api-access-k7s6f\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:16.938687 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.938514 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20df45c3-27b5-4510-883c-3e9354e85fbe-openshift-service-ca\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:16.938687 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:28:16.938575 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 31 14:28:16.938687 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:28:16.938643 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20df45c3-27b5-4510-883c-3e9354e85fbe-maas-api-tls podName:20df45c3-27b5-4510-883c-3e9354e85fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:28:17.438625209 +0000 UTC m=+962.947004272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20df45c3-27b5-4510-883c-3e9354e85fbe-maas-api-tls") pod "maas-api-7c89d6c77-5stlh" (UID: "20df45c3-27b5-4510-883c-3e9354e85fbe") : secret "maas-api-serving-cert" not found Jul 31 14:28:16.939221 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.939202 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20df45c3-27b5-4510-883c-3e9354e85fbe-openshift-service-ca\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:16.949768 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:16.949736 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7s6f\" (UniqueName: \"kubernetes.io/projected/20df45c3-27b5-4510-883c-3e9354e85fbe-kube-api-access-k7s6f\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:17.035192 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.035147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:17.203881 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.203766 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:17.292783 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.292749 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:17.298122 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.298093 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57"] Jul 31 14:28:17.304098 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.304075 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:17.309942 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.309894 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 31 14:28:17.310055 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.309967 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-swct6\"" Jul 31 14:28:17.315187 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.315165 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57"] Jul 31 14:28:17.334735 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.334705 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mgmqg"] Jul 31 14:28:17.339112 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.339094 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:17.348067 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.348041 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mgmqg"] Jul 31 14:28:17.443835 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.443796 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcj8h\" (UniqueName: \"kubernetes.io/projected/66557854-208e-4687-a3dd-33a89c462995-kube-api-access-zcj8h\") pod \"payload-pre-processing-687dbb6cf6-ktn57\" (UID: \"66557854-208e-4687-a3dd-33a89c462995\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:17.444033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.443843 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20df45c3-27b5-4510-883c-3e9354e85fbe-maas-api-tls\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:17.444033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.443935 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb0610d-5aa3-463d-a582-a3ba5a58c207-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mgmqg\" (UID: \"ddb0610d-5aa3-463d-a582-a3ba5a58c207\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:17.444033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.443971 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcplp\" (UniqueName: \"kubernetes.io/projected/ddb0610d-5aa3-463d-a582-a3ba5a58c207-kube-api-access-tcplp\") pod \"payload-processing-5cc6cd444b-mgmqg\" (UID: \"ddb0610d-5aa3-463d-a582-a3ba5a58c207\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:17.444159 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.444056 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66557854-208e-4687-a3dd-33a89c462995-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-ktn57\" (UID: \"66557854-208e-4687-a3dd-33a89c462995\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:17.446291 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.446264 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20df45c3-27b5-4510-883c-3e9354e85fbe-maas-api-tls\") pod \"maas-api-7c89d6c77-5stlh\" (UID: \"20df45c3-27b5-4510-883c-3e9354e85fbe\") " pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:17.545709 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.545609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb0610d-5aa3-463d-a582-a3ba5a58c207-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mgmqg\" (UID: \"ddb0610d-5aa3-463d-a582-a3ba5a58c207\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:17.545709 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.545655 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcplp\" (UniqueName: \"kubernetes.io/projected/ddb0610d-5aa3-463d-a582-a3ba5a58c207-kube-api-access-tcplp\") pod \"payload-processing-5cc6cd444b-mgmqg\" (UID: \"ddb0610d-5aa3-463d-a582-a3ba5a58c207\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:17.545953 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.545779 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66557854-208e-4687-a3dd-33a89c462995-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-ktn57\" (UID: \"66557854-208e-4687-a3dd-33a89c462995\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:17.545953 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.545896 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcj8h\" (UniqueName: \"kubernetes.io/projected/66557854-208e-4687-a3dd-33a89c462995-kube-api-access-zcj8h\") pod \"payload-pre-processing-687dbb6cf6-ktn57\" (UID: \"66557854-208e-4687-a3dd-33a89c462995\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:17.546392 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.546372 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb0610d-5aa3-463d-a582-a3ba5a58c207-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mgmqg\" (UID: \"ddb0610d-5aa3-463d-a582-a3ba5a58c207\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:17.546519 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.546500 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66557854-208e-4687-a3dd-33a89c462995-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-ktn57\" (UID: \"66557854-208e-4687-a3dd-33a89c462995\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:17.553922 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.553881 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcplp\" (UniqueName: \"kubernetes.io/projected/ddb0610d-5aa3-463d-a582-a3ba5a58c207-kube-api-access-tcplp\") pod \"payload-processing-5cc6cd444b-mgmqg\" (UID: \"ddb0610d-5aa3-463d-a582-a3ba5a58c207\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:17.554033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.554012 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcj8h\" (UniqueName: \"kubernetes.io/projected/66557854-208e-4687-a3dd-33a89c462995-kube-api-access-zcj8h\") pod \"payload-pre-processing-687dbb6cf6-ktn57\" (UID: \"66557854-208e-4687-a3dd-33a89c462995\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:17.616146 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.616114 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:17.639441 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.639413 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:17.652604 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.652554 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:17.777889 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.777778 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57"] Jul 31 14:28:17.782508 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:28:17.782473 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66557854_208e_4687_a3dd_33a89c462995.slice/crio-74e8ac7f10b6a60e0d8cff4984262ee488914b85fff68c8a4e7d1fb4be7ed7de WatchSource:0}: Error finding container 74e8ac7f10b6a60e0d8cff4984262ee488914b85fff68c8a4e7d1fb4be7ed7de: Status 404 returned error can't find the container with id 74e8ac7f10b6a60e0d8cff4984262ee488914b85fff68c8a4e7d1fb4be7ed7de Jul 31 14:28:17.784677 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.784654 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:28:17.805233 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.805210 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh"] Jul 31 14:28:17.808154 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:28:17.808118 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20df45c3_27b5_4510_883c_3e9354e85fbe.slice/crio-ac2946cdee8fb64b0fa066416c67b3101170afed0b55866d1fbb26ba3be2defa WatchSource:0}: Error finding container ac2946cdee8fb64b0fa066416c67b3101170afed0b55866d1fbb26ba3be2defa: Status 404 returned error can't find the container with id ac2946cdee8fb64b0fa066416c67b3101170afed0b55866d1fbb26ba3be2defa Jul 31 14:28:17.829171 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.827131 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mgmqg"] Jul 31 14:28:17.832053 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:28:17.832029 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb0610d_5aa3_463d_a582_a3ba5a58c207.slice/crio-a36bd15ae70879c00b4c5a0cee07c32216637dd04d288fd56bec25b5d0372bd0 WatchSource:0}: Error finding container a36bd15ae70879c00b4c5a0cee07c32216637dd04d288fd56bec25b5d0372bd0: Status 404 returned error can't find the container with id a36bd15ae70879c00b4c5a0cee07c32216637dd04d288fd56bec25b5d0372bd0 Jul 31 14:28:17.833598 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.833559 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" event={"ID":"20df45c3-27b5-4510-883c-3e9354e85fbe","Type":"ContainerStarted","Data":"ac2946cdee8fb64b0fa066416c67b3101170afed0b55866d1fbb26ba3be2defa"} Jul 31 14:28:17.834741 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:17.834717 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" event={"ID":"66557854-208e-4687-a3dd-33a89c462995","Type":"ContainerStarted","Data":"74e8ac7f10b6a60e0d8cff4984262ee488914b85fff68c8a4e7d1fb4be7ed7de"} Jul 31 14:28:18.844982 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:18.844941 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" event={"ID":"ddb0610d-5aa3-463d-a582-a3ba5a58c207","Type":"ContainerStarted","Data":"a36bd15ae70879c00b4c5a0cee07c32216637dd04d288fd56bec25b5d0372bd0"} Jul 31 14:28:20.862273 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.862236 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" event={"ID":"20df45c3-27b5-4510-883c-3e9354e85fbe","Type":"ContainerStarted","Data":"899c989706444ed0c80cef2c7312239b2200e88714733613aa7bbb490379d70d"} Jul 31 14:28:20.862827 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.862350 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:20.863993 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.863966 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" event={"ID":"ddb0610d-5aa3-463d-a582-a3ba5a58c207","Type":"ContainerStarted","Data":"6bffab5d51d5f7dbfe958b056afca3ca36c0ffc44f1f70a1550318d8532f3285"} Jul 31 14:28:20.864130 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.864064 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:20.868499 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.868476 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" event={"ID":"66557854-208e-4687-a3dd-33a89c462995","Type":"ContainerStarted","Data":"54fb07bb1636c3ff9792ec6bb4e9b4c2ed0382e0d35cae0fb0782f2baff7af77"} Jul 31 14:28:20.868591 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.868550 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:20.886406 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.886366 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" podStartSLOduration=2.8368277710000003 podStartE2EDuration="4.886351624s" totalImagesPullingTime="2.049523853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:28:20.885400942 +0000 UTC m=+966.393780008" watchObservedRunningTime="2026-07-31 14:28:20.886351624 +0000 UTC m=+966.394730690" Jul 31 14:28:20.902886 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.902834 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" podStartSLOduration=1.866815356 podStartE2EDuration="3.902817749s" totalImagesPullingTime="2.036002393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:28:20.900258965 +0000 UTC m=+966.408638031" watchObservedRunningTime="2026-07-31 14:28:20.902817749 +0000 UTC m=+966.411196819" Jul 31 14:28:20.914183 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:20.914141 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" podStartSLOduration=1.839308238 podStartE2EDuration="3.914127234s" totalImagesPullingTime="2.074818996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:28:20.913775685 +0000 UTC m=+966.422154750" watchObservedRunningTime="2026-07-31 14:28:20.914127234 +0000 UTC m=+966.422506299" Jul 31 14:28:26.878606 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:26.878572 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c89d6c77-5stlh" Jul 31 14:28:27.995780 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:27.995746 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:28.090390 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:28.090333 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:31.874810 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:31.874776 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-mgmqg" Jul 31 14:28:31.875253 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:31.874844 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-ktn57" Jul 31 14:28:42.685665 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.685627 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29"] Jul 31 14:28:42.689018 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.688995 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.692999 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.692973 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 31 14:28:42.692999 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.692994 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7xdgk\"" Jul 31 14:28:42.693244 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.693228 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 31 14:28:42.693358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.693342 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 31 14:28:42.699529 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.699505 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29"] Jul 31 14:28:42.709075 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.709047 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:42.754143 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.754120 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.754248 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.754189 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.754293 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.754253 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.754329 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.754299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d08f180-d899-4391-98ca-b0e60b6fe968-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.754364 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.754330 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqgdp\" (UniqueName: \"kubernetes.io/projected/7d08f180-d899-4391-98ca-b0e60b6fe968-kube-api-access-nqgdp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.754461 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.754442 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.855823 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.855787 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqgdp\" (UniqueName: \"kubernetes.io/projected/7d08f180-d899-4391-98ca-b0e60b6fe968-kube-api-access-nqgdp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.856016 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.855860 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.856016 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.855894 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.856016 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.855959 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.856016 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.855988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.856187 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.856029 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d08f180-d899-4391-98ca-b0e60b6fe968-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.856312 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.856289 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.856363 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.856328 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.856363 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.856354 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.858293 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.858270 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d08f180-d899-4391-98ca-b0e60b6fe968-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.858532 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.858516 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d08f180-d899-4391-98ca-b0e60b6fe968-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:42.863280 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:42.863256 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqgdp\" (UniqueName: \"kubernetes.io/projected/7d08f180-d899-4391-98ca-b0e60b6fe968-kube-api-access-nqgdp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29\" (UID: \"7d08f180-d899-4391-98ca-b0e60b6fe968\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:43.002540 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:43.002454 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:43.139685 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:43.139657 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29"] Jul 31 14:28:43.141375 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:28:43.141347 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d08f180_d899_4391_98ca_b0e60b6fe968.slice/crio-9be17476e101b0e36b21fafeb15c387fd3b30f3b42ad22d9948889aa4ec257d6 WatchSource:0}: Error finding container 9be17476e101b0e36b21fafeb15c387fd3b30f3b42ad22d9948889aa4ec257d6: Status 404 returned error can't find the container with id 9be17476e101b0e36b21fafeb15c387fd3b30f3b42ad22d9948889aa4ec257d6 Jul 31 14:28:43.984919 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:43.984867 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" event={"ID":"7d08f180-d899-4391-98ca-b0e60b6fe968","Type":"ContainerStarted","Data":"9be17476e101b0e36b21fafeb15c387fd3b30f3b42ad22d9948889aa4ec257d6"} Jul 31 14:28:45.996646 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:45.996609 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" event={"ID":"7d08f180-d899-4391-98ca-b0e60b6fe968","Type":"ContainerStarted","Data":"e9dc63bbcf16c33e1fab7d58f2d8de3a6e825061aa194b66a4c6cc660ff462c8"} Jul 31 14:28:46.013179 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:46.013136 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" podStartSLOduration=1.246962168 podStartE2EDuration="4.01311798s" totalImagesPullingTime="2.766155812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:28:46.012501012 +0000 UTC m=+991.520880080" watchObservedRunningTime="2026-07-31 14:28:46.01311798 +0000 UTC m=+991.521497045" Jul 31 14:28:49.894948 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:49.894894 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:53.003387 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:53.003353 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:53.003860 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:53.003415 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:53.015803 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:53.015778 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:53.042107 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:53.042084 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29" Jul 31 14:28:55.630587 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:55.630542 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:28:55.731776 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:28:55.731722 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:29:04.297931 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:04.297868 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:29:15.082593 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.082560 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs"] Jul 31 14:29:15.085113 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.085097 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.087760 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.087738 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 31 14:29:15.093854 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.093830 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs"] Jul 31 14:29:15.170247 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.170206 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:29:15.243516 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.243482 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.243678 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.243549 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2sb5\" (UniqueName: \"kubernetes.io/projected/c0503050-f883-4e31-bcce-c0f0bf64c480-kube-api-access-z2sb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.243678 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.243607 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.243678 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.243630 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.243847 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.243711 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0503050-f883-4e31-bcce-c0f0bf64c480-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.243847 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.243773 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345010 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.344890 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0503050-f883-4e31-bcce-c0f0bf64c480-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345010 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.344960 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345010 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.344988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345320 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.345141 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2sb5\" (UniqueName: \"kubernetes.io/projected/c0503050-f883-4e31-bcce-c0f0bf64c480-kube-api-access-z2sb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345320 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.345286 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345448 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.345326 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345516 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.345462 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345680 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.345645 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.345680 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.345664 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.347333 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.347313 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0503050-f883-4e31-bcce-c0f0bf64c480-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.347439 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.347422 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0503050-f883-4e31-bcce-c0f0bf64c480-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.353464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.353442 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2sb5\" (UniqueName: \"kubernetes.io/projected/c0503050-f883-4e31-bcce-c0f0bf64c480-kube-api-access-z2sb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs\" (UID: \"c0503050-f883-4e31-bcce-c0f0bf64c480\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.395662 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.395628 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:15.525740 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:15.525708 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs"] Jul 31 14:29:15.526768 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:29:15.526738 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0503050_f883_4e31_bcce_c0f0bf64c480.slice/crio-2cc144d3833ae2546ee62fd2c902d96428ed91f21d25b3039fd6143eaaa5b07d WatchSource:0}: Error finding container 2cc144d3833ae2546ee62fd2c902d96428ed91f21d25b3039fd6143eaaa5b07d: Status 404 returned error can't find the container with id 2cc144d3833ae2546ee62fd2c902d96428ed91f21d25b3039fd6143eaaa5b07d Jul 31 14:29:16.148271 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:16.148241 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" event={"ID":"c0503050-f883-4e31-bcce-c0f0bf64c480","Type":"ContainerStarted","Data":"fc45c6f3bbf4622cb7ab89c0d9949a0bfcf51ff279043c82f6b52edd6b95dca7"} Jul 31 14:29:16.148644 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:16.148292 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" event={"ID":"c0503050-f883-4e31-bcce-c0f0bf64c480","Type":"ContainerStarted","Data":"2cc144d3833ae2546ee62fd2c902d96428ed91f21d25b3039fd6143eaaa5b07d"} Jul 31 14:29:16.171374 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:16.171319 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" podStartSLOduration=1.0014599 podStartE2EDuration="1.171305877s" totalImagesPullingTime="169.845977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:29:16.168707892 +0000 UTC m=+1021.677086961" watchObservedRunningTime="2026-07-31 14:29:16.171305877 +0000 UTC m=+1021.679684942" Jul 31 14:29:18.491085 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:18.491040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:29:25.396847 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:25.396807 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:25.397392 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:25.396881 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:25.409858 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:25.409828 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:26.213542 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:26.213515 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs" Jul 31 14:29:32.871684 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.871638 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz"] Jul 31 14:29:32.876943 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.876922 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:32.879855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.879829 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 31 14:29:32.888146 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.888120 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz"] Jul 31 14:29:32.904076 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.904048 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:29:32.985501 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.985470 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7690e4-ba60-49c8-82ef-5ff408c248ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:32.985647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.985511 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:32.985647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.985536 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:32.985647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.985553 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhq8\" (UniqueName: \"kubernetes.io/projected/7f7690e4-ba60-49c8-82ef-5ff408c248ac-kube-api-access-wzhq8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:32.985647 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.985617 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:32.985831 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:32.985669 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.086812 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.086777 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.086812 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.086811 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhq8\" (UniqueName: \"kubernetes.io/projected/7f7690e4-ba60-49c8-82ef-5ff408c248ac-kube-api-access-wzhq8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.087098 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.086945 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.087098 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.087021 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.087098 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.087071 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7690e4-ba60-49c8-82ef-5ff408c248ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.087267 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.087107 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.087419 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.087399 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.087472 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.087432 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.087472 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.087452 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.089197 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.089176 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7690e4-ba60-49c8-82ef-5ff408c248ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.089581 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.089559 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7690e4-ba60-49c8-82ef-5ff408c248ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.096272 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.096249 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhq8\" (UniqueName: \"kubernetes.io/projected/7f7690e4-ba60-49c8-82ef-5ff408c248ac-kube-api-access-wzhq8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5bljz\" (UID: \"7f7690e4-ba60-49c8-82ef-5ff408c248ac\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.187340 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.187275 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:33.315980 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:33.315956 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz"] Jul 31 14:29:33.317423 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:29:33.317397 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f7690e4_ba60_49c8_82ef_5ff408c248ac.slice/crio-7ca35525371d4467671926d6364102ef44f5acf097c42ea06de88300f7f85300 WatchSource:0}: Error finding container 7ca35525371d4467671926d6364102ef44f5acf097c42ea06de88300f7f85300: Status 404 returned error can't find the container with id 7ca35525371d4467671926d6364102ef44f5acf097c42ea06de88300f7f85300 Jul 31 14:29:34.248068 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:34.248034 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" event={"ID":"7f7690e4-ba60-49c8-82ef-5ff408c248ac","Type":"ContainerStarted","Data":"203bf8a20c90b4c1ea1646de4ddc2f53041658212bcefd6a28be7b36088ed8df"} Jul 31 14:29:34.248447 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:34.248079 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" event={"ID":"7f7690e4-ba60-49c8-82ef-5ff408c248ac","Type":"ContainerStarted","Data":"7ca35525371d4467671926d6364102ef44f5acf097c42ea06de88300f7f85300"} Jul 31 14:29:34.264814 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:34.264769 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" podStartSLOduration=2.084975079 podStartE2EDuration="2.26475548s" totalImagesPullingTime="179.780401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:29:34.26417519 +0000 UTC m=+1039.772554256" watchObservedRunningTime="2026-07-31 14:29:34.26475548 +0000 UTC m=+1039.773134555" Jul 31 14:29:43.188413 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:43.188375 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:43.188849 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:43.188429 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:43.200806 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:43.200777 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:43.305134 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:43.305105 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5bljz" Jul 31 14:29:52.580191 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:29:52.580151 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:30:30.376453 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.376371 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56bdb4749c-qjp6l"] Jul 31 14:30:30.379975 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.379955 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56bdb4749c-qjp6l" Jul 31 14:30:30.383814 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.383792 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wx7kw\"" Jul 31 14:30:30.383952 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.383842 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 31 14:30:30.387694 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.387436 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56bdb4749c-qjp6l"] Jul 31 14:30:30.488492 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.488455 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6362da31-9896-4393-920c-28f619a89294-tls-cert\") pod \"authorino-56bdb4749c-qjp6l\" (UID: \"6362da31-9896-4393-920c-28f619a89294\") " pod="kuadrant-system/authorino-56bdb4749c-qjp6l" Jul 31 14:30:30.488680 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.488534 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2zh\" (UniqueName: \"kubernetes.io/projected/6362da31-9896-4393-920c-28f619a89294-kube-api-access-dj2zh\") pod \"authorino-56bdb4749c-qjp6l\" (UID: \"6362da31-9896-4393-920c-28f619a89294\") " pod="kuadrant-system/authorino-56bdb4749c-qjp6l" Jul 31 14:30:30.589330 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.589291 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6362da31-9896-4393-920c-28f619a89294-tls-cert\") pod \"authorino-56bdb4749c-qjp6l\" (UID: \"6362da31-9896-4393-920c-28f619a89294\") " pod="kuadrant-system/authorino-56bdb4749c-qjp6l" Jul 31 14:30:30.589512 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.589448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2zh\" (UniqueName: \"kubernetes.io/projected/6362da31-9896-4393-920c-28f619a89294-kube-api-access-dj2zh\") pod \"authorino-56bdb4749c-qjp6l\" (UID: \"6362da31-9896-4393-920c-28f619a89294\") " pod="kuadrant-system/authorino-56bdb4749c-qjp6l" Jul 31 14:30:30.591834 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.591808 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6362da31-9896-4393-920c-28f619a89294-tls-cert\") pod \"authorino-56bdb4749c-qjp6l\" (UID: \"6362da31-9896-4393-920c-28f619a89294\") " pod="kuadrant-system/authorino-56bdb4749c-qjp6l" Jul 31 14:30:30.597563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.597542 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2zh\" (UniqueName: \"kubernetes.io/projected/6362da31-9896-4393-920c-28f619a89294-kube-api-access-dj2zh\") pod \"authorino-56bdb4749c-qjp6l\" (UID: \"6362da31-9896-4393-920c-28f619a89294\") " pod="kuadrant-system/authorino-56bdb4749c-qjp6l" Jul 31 14:30:30.691210 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.691149 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56bdb4749c-qjp6l" Jul 31 14:30:30.820154 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:30.820124 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56bdb4749c-qjp6l"] Jul 31 14:30:30.821541 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:30:30.821514 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6362da31_9896_4393_920c_28f619a89294.slice/crio-4f4ab7b1a148570f4ca65ad6e94acd6e587e9631b4f9898fe2f6298531ed4bd3 WatchSource:0}: Error finding container 4f4ab7b1a148570f4ca65ad6e94acd6e587e9631b4f9898fe2f6298531ed4bd3: Status 404 returned error can't find the container with id 4f4ab7b1a148570f4ca65ad6e94acd6e587e9631b4f9898fe2f6298531ed4bd3 Jul 31 14:30:31.539414 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:31.539369 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56bdb4749c-qjp6l" event={"ID":"6362da31-9896-4393-920c-28f619a89294","Type":"ContainerStarted","Data":"4f4ab7b1a148570f4ca65ad6e94acd6e587e9631b4f9898fe2f6298531ed4bd3"} Jul 31 14:30:32.546672 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:32.546624 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56bdb4749c-qjp6l" event={"ID":"6362da31-9896-4393-920c-28f619a89294","Type":"ContainerStarted","Data":"f9109dfd1a9ede3bba20d7adde97cafcdc4e267b694767ee1992647315c5fa00"} Jul 31 14:30:32.564819 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:32.564763 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56bdb4749c-qjp6l" podStartSLOduration=1.848213041 podStartE2EDuration="2.564747791s" totalImagesPullingTime="716.53475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:30:32.562623273 +0000 UTC m=+1098.071002331" watchObservedRunningTime="2026-07-31 14:30:32.564747791 +0000 UTC m=+1098.073126857" Jul 31 14:30:33.084804 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:33.084770 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:30:33.185008 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:33.184967 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:30:34.210836 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:34.210799 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:30:34.315839 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:30:34.315799 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:31:21.846471 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:21.846438 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f846468f4-nqqzl"] Jul 31 14:31:21.847122 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:21.846677 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f846468f4-nqqzl" podUID="71733483-ded1-487a-9106-aa92b9fe849b" containerName="manager" containerID="cri-o://4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5" gracePeriod=30 Jul 31 14:31:22.100610 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.100548 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:31:22.140604 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.140570 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71733483-ded1-487a-9106-aa92b9fe849b-webhook-cert\") pod \"71733483-ded1-487a-9106-aa92b9fe849b\" (UID: \"71733483-ded1-487a-9106-aa92b9fe849b\") " Jul 31 14:31:22.140779 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.140697 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqpmr\" (UniqueName: \"kubernetes.io/projected/71733483-ded1-487a-9106-aa92b9fe849b-kube-api-access-rqpmr\") pod \"71733483-ded1-487a-9106-aa92b9fe849b\" (UID: \"71733483-ded1-487a-9106-aa92b9fe849b\") " Jul 31 14:31:22.142384 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.142357 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71733483-ded1-487a-9106-aa92b9fe849b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "71733483-ded1-487a-9106-aa92b9fe849b" (UID: "71733483-ded1-487a-9106-aa92b9fe849b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:31:22.142553 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.142535 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71733483-ded1-487a-9106-aa92b9fe849b-kube-api-access-rqpmr" (OuterVolumeSpecName: "kube-api-access-rqpmr") pod "71733483-ded1-487a-9106-aa92b9fe849b" (UID: "71733483-ded1-487a-9106-aa92b9fe849b"). InnerVolumeSpecName "kube-api-access-rqpmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:31:22.242233 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.242205 2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71733483-ded1-487a-9106-aa92b9fe849b-webhook-cert\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:31:22.242233 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.242236 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqpmr\" (UniqueName: \"kubernetes.io/projected/71733483-ded1-487a-9106-aa92b9fe849b-kube-api-access-rqpmr\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:31:22.820383 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.820341 2558 generic.go:356] "Generic (PLEG): container finished" podID="71733483-ded1-487a-9106-aa92b9fe849b" containerID="4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5" exitCode=0 Jul 31 14:31:22.820563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.820389 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-nqqzl" Jul 31 14:31:22.820563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.820426 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-nqqzl" event={"ID":"71733483-ded1-487a-9106-aa92b9fe849b","Type":"ContainerDied","Data":"4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5"} Jul 31 14:31:22.820563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.820497 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-nqqzl" event={"ID":"71733483-ded1-487a-9106-aa92b9fe849b","Type":"ContainerDied","Data":"4009234066a0cf8a4781fb53bf888da1b726e92872223bec94db02c6b810204f"} Jul 31 14:31:22.820563 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.820525 2558 scope.go:117] "RemoveContainer" containerID="4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5" Jul 31 14:31:22.831551 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.831531 2558 scope.go:117] "RemoveContainer" containerID="4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5" Jul 31 14:31:22.831838 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:31:22.831811 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5\": container with ID starting with 4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5 not found: ID does not exist" containerID="4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5" Jul 31 14:31:22.831951 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.831845 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5"} err="failed to get container status \"4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5\": rpc error: code = NotFound desc = could not find container \"4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5\": container with ID starting with 4631a1fb0669ad8797cca85cc6ba62e3fa2c87d735a3405f23bc07f3348888b5 not found: ID does not exist" Jul 31 14:31:22.843609 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.843575 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f846468f4-nqqzl"] Jul 31 14:31:22.845709 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.845690 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f846468f4-nqqzl"] Jul 31 14:31:22.983476 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:22.983443 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71733483-ded1-487a-9106-aa92b9fe849b" path="/var/lib/kubelet/pods/71733483-ded1-487a-9106-aa92b9fe849b/volumes" Jul 31 14:31:34.718987 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:34.718948 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:31:34.819628 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:34.819598 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:31:59.543438 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:31:59.543394 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:32:09.686372 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:09.686336 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:32:23.682989 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.682952 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f846468f4-kdjlx"] Jul 31 14:32:23.683369 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.683314 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71733483-ded1-487a-9106-aa92b9fe849b" containerName="manager" Jul 31 14:32:23.683369 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.683325 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="71733483-ded1-487a-9106-aa92b9fe849b" containerName="manager" Jul 31 14:32:23.683442 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.683387 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="71733483-ded1-487a-9106-aa92b9fe849b" containerName="manager" Jul 31 14:32:23.686661 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.686645 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:23.690759 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.690733 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rvdts\"" Jul 31 14:32:23.690759 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.690752 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 14:32:23.690953 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.690790 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 14:32:23.695555 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.695532 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f846468f4-kdjlx"] Jul 31 14:32:23.774400 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.774368 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-webhook-cert\") pod \"maas-controller-f846468f4-kdjlx\" (UID: \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\") " pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:23.774564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.774411 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qt8\" (UniqueName: \"kubernetes.io/projected/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-kube-api-access-88qt8\") pod \"maas-controller-f846468f4-kdjlx\" (UID: \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\") " pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:23.875717 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.875680 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-webhook-cert\") pod \"maas-controller-f846468f4-kdjlx\" (UID: \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\") " pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:23.875890 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.875726 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qt8\" (UniqueName: \"kubernetes.io/projected/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-kube-api-access-88qt8\") pod \"maas-controller-f846468f4-kdjlx\" (UID: \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\") " pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:23.878128 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.878109 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-webhook-cert\") pod \"maas-controller-f846468f4-kdjlx\" (UID: \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\") " pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:23.884205 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.884180 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qt8\" (UniqueName: \"kubernetes.io/projected/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-kube-api-access-88qt8\") pod \"maas-controller-f846468f4-kdjlx\" (UID: \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\") " pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:23.998522 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:23.998440 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:24.133686 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:24.133662 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f846468f4-kdjlx"] Jul 31 14:32:24.134691 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:32:24.134663 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ea4795_7fc4_4144_9ac0_a3a3aacd8639.slice/crio-fe17f02a97e0001a1759e0394543efc3f1df919407e0275a40c7b2601410bb4d WatchSource:0}: Error finding container fe17f02a97e0001a1759e0394543efc3f1df919407e0275a40c7b2601410bb4d: Status 404 returned error can't find the container with id fe17f02a97e0001a1759e0394543efc3f1df919407e0275a40c7b2601410bb4d Jul 31 14:32:25.140263 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:25.140224 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-kdjlx" event={"ID":"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639","Type":"ContainerStarted","Data":"ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4"} Jul 31 14:32:25.140660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:25.140282 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-kdjlx" event={"ID":"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639","Type":"ContainerStarted","Data":"fe17f02a97e0001a1759e0394543efc3f1df919407e0275a40c7b2601410bb4d"} Jul 31 14:32:25.140660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:25.140332 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:25.158961 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:25.158899 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f846468f4-kdjlx" podStartSLOduration=1.78504824 podStartE2EDuration="2.158886272s" totalImagesPullingTime="373.838032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:32:25.158131919 +0000 UTC m=+1210.666510987" watchObservedRunningTime="2026-07-31 14:32:25.158886272 +0000 UTC m=+1210.667265338" Jul 31 14:32:36.146879 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:36.146847 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:40.689914 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:40.689876 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:32:40.797291 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:40.797256 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f846468f4-kdjlx"] Jul 31 14:32:40.797517 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:40.797484 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f846468f4-kdjlx" podUID="b7ea4795-7fc4-4144-9ac0-a3a3aacd8639" containerName="manager" containerID="cri-o://ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4" gracePeriod=30 Jul 31 14:32:41.046445 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.046417 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:41.127696 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.127664 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-webhook-cert\") pod \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\" (UID: \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\") " Jul 31 14:32:41.127855 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.127740 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88qt8\" (UniqueName: \"kubernetes.io/projected/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-kube-api-access-88qt8\") pod \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\" (UID: \"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639\") " Jul 31 14:32:41.129573 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.129543 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b7ea4795-7fc4-4144-9ac0-a3a3aacd8639" (UID: "b7ea4795-7fc4-4144-9ac0-a3a3aacd8639"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:32:41.129744 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.129723 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-kube-api-access-88qt8" (OuterVolumeSpecName: "kube-api-access-88qt8") pod "b7ea4795-7fc4-4144-9ac0-a3a3aacd8639" (UID: "b7ea4795-7fc4-4144-9ac0-a3a3aacd8639"). InnerVolumeSpecName "kube-api-access-88qt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:32:41.221747 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.221661 2558 generic.go:356] "Generic (PLEG): container finished" podID="b7ea4795-7fc4-4144-9ac0-a3a3aacd8639" containerID="ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4" exitCode=0 Jul 31 14:32:41.221747 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.221719 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-kdjlx" Jul 31 14:32:41.222012 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.221750 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-kdjlx" event={"ID":"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639","Type":"ContainerDied","Data":"ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4"} Jul 31 14:32:41.222012 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.221843 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-kdjlx" event={"ID":"b7ea4795-7fc4-4144-9ac0-a3a3aacd8639","Type":"ContainerDied","Data":"fe17f02a97e0001a1759e0394543efc3f1df919407e0275a40c7b2601410bb4d"} Jul 31 14:32:41.222012 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.221874 2558 scope.go:117] "RemoveContainer" containerID="ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4" Jul 31 14:32:41.228361 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.228341 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qt8\" (UniqueName: \"kubernetes.io/projected/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-kube-api-access-88qt8\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:32:41.228455 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.228362 2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639-webhook-cert\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:32:41.231097 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.231081 2558 scope.go:117] "RemoveContainer" containerID="ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4" Jul 31 14:32:41.231329 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:32:41.231311 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4\": container with ID starting with ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4 not found: ID does not exist" containerID="ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4" Jul 31 14:32:41.231374 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.231337 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4"} err="failed to get container status \"ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4\": rpc error: code = NotFound desc = could not find container \"ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4\": container with ID starting with ceb6eb6ee6d5914508aa706b47834b86d2f084992a0e187667bcdcd65f7722a4 not found: ID does not exist" Jul 31 14:32:41.244851 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.244820 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f846468f4-kdjlx"] Jul 31 14:32:41.249663 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:41.249627 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f846468f4-kdjlx"] Jul 31 14:32:42.063671 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.063631 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f846468f4-5p9p9"] Jul 31 14:32:42.064253 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.064232 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7ea4795-7fc4-4144-9ac0-a3a3aacd8639" containerName="manager" Jul 31 14:32:42.064334 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.064256 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7ea4795-7fc4-4144-9ac0-a3a3aacd8639" containerName="manager" Jul 31 14:32:42.064388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.064343 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7ea4795-7fc4-4144-9ac0-a3a3aacd8639" containerName="manager" Jul 31 14:32:42.068682 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.068661 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:42.073104 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.073082 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rvdts\"" Jul 31 14:32:42.073219 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.073101 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 31 14:32:42.073219 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.073119 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 31 14:32:42.077476 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.077454 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f846468f4-5p9p9"] Jul 31 14:32:42.131596 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.131552 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:32:42.136636 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.136616 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22bf2228-9058-4118-8898-9b0635d27b3f-webhook-cert\") pod \"maas-controller-f846468f4-5p9p9\" (UID: \"22bf2228-9058-4118-8898-9b0635d27b3f\") " pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:42.136706 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.136646 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97vq\" (UniqueName: \"kubernetes.io/projected/22bf2228-9058-4118-8898-9b0635d27b3f-kube-api-access-x97vq\") pod \"maas-controller-f846468f4-5p9p9\" (UID: \"22bf2228-9058-4118-8898-9b0635d27b3f\") " pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:42.237749 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.237705 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22bf2228-9058-4118-8898-9b0635d27b3f-webhook-cert\") pod \"maas-controller-f846468f4-5p9p9\" (UID: \"22bf2228-9058-4118-8898-9b0635d27b3f\") " pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:42.237749 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.237752 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97vq\" (UniqueName: \"kubernetes.io/projected/22bf2228-9058-4118-8898-9b0635d27b3f-kube-api-access-x97vq\") pod \"maas-controller-f846468f4-5p9p9\" (UID: \"22bf2228-9058-4118-8898-9b0635d27b3f\") " pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:42.240289 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.240268 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22bf2228-9058-4118-8898-9b0635d27b3f-webhook-cert\") pod \"maas-controller-f846468f4-5p9p9\" (UID: \"22bf2228-9058-4118-8898-9b0635d27b3f\") " pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:42.245780 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.245755 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97vq\" (UniqueName: \"kubernetes.io/projected/22bf2228-9058-4118-8898-9b0635d27b3f-kube-api-access-x97vq\") pod \"maas-controller-f846468f4-5p9p9\" (UID: \"22bf2228-9058-4118-8898-9b0635d27b3f\") " pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:42.380171 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.380145 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:42.511705 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.511679 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f846468f4-5p9p9"] Jul 31 14:32:42.513359 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:32:42.513333 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bf2228_9058_4118_8898_9b0635d27b3f.slice/crio-4b80f2aa0bc9a98c69750567abde55add6b54510360f9286935b23f1792423eb WatchSource:0}: Error finding container 4b80f2aa0bc9a98c69750567abde55add6b54510360f9286935b23f1792423eb: Status 404 returned error can't find the container with id 4b80f2aa0bc9a98c69750567abde55add6b54510360f9286935b23f1792423eb Jul 31 14:32:42.983278 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:42.983245 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7ea4795-7fc4-4144-9ac0-a3a3aacd8639" path="/var/lib/kubelet/pods/b7ea4795-7fc4-4144-9ac0-a3a3aacd8639/volumes" Jul 31 14:32:43.237488 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:43.237386 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-5p9p9" event={"ID":"22bf2228-9058-4118-8898-9b0635d27b3f","Type":"ContainerStarted","Data":"cc86f81cb8b74ad182c792fee55250c49c520cb52419fff932b009435d8cedc3"} Jul 31 14:32:43.237488 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:43.237441 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f846468f4-5p9p9" event={"ID":"22bf2228-9058-4118-8898-9b0635d27b3f","Type":"ContainerStarted","Data":"4b80f2aa0bc9a98c69750567abde55add6b54510360f9286935b23f1792423eb"} Jul 31 14:32:43.237488 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:43.237463 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:32:43.255672 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:43.255626 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f846468f4-5p9p9" podStartSLOduration=0.709579473 podStartE2EDuration="1.255611717s" totalImagesPullingTime="546.032244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:32:43.253274868 +0000 UTC m=+1228.761653937" watchObservedRunningTime="2026-07-31 14:32:43.255611717 +0000 UTC m=+1228.763990804" Jul 31 14:32:54.244490 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:32:54.244456 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f846468f4-5p9p9" Jul 31 14:33:51.655295 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:51.655259 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:51.751819 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:51.751780 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:51.891599 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:51.891559 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:57.550676 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:57.550641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:57.613738 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:57.613703 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:57.705043 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:57.705004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:57.811522 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:57.811423 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:57.923754 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:57.923721 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:59.204474 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:59.204420 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:33:59.305856 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:33:59.305810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:34:03.498702 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:34:03.498647 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:34:03.598980 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:34:03.598948 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:34:59.434152 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:34:59.434108 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:36:04.221678 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:36:04.221645 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:36:04.276558 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:36:04.276526 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:38:25.483352 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:38:25.483290 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:41:42.345912 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:41:42.345865 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:42:31.286477 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:42:31.286440 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:43:49.803820 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:43:49.803779 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:44:14.616077 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:14.616029 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:44:47.681279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.681227 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg"] Jul 31 14:44:47.684817 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.684799 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.687679 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.687655 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6b2a4720-kserve-self-signed-certs\"" Jul 31 14:44:47.695034 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.695010 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg"] Jul 31 14:44:47.698953 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.698931 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-model-cache\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.699056 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.698989 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-dshm\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.699056 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.699028 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tls-certs\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.699132 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.699054 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wwb\" (UniqueName: \"kubernetes.io/projected/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-kube-api-access-w9wwb\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.699132 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.699100 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tmp-dir\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.699132 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.699120 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-home\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.708857 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.708816 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:44:47.799721 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.799681 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wwb\" (UniqueName: \"kubernetes.io/projected/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-kube-api-access-w9wwb\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.799892 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.799730 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tmp-dir\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.799892 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.799749 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-home\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.799892 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.799782 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-model-cache\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.799892 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.799842 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-dshm\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.799892 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.799872 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tls-certs\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.800197 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.800185 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tmp-dir\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.800258 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.800243 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-home\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.800314 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.800258 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-model-cache\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.802217 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.802198 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-dshm\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.802502 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.802481 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tls-certs\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.807380 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.807355 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wwb\" (UniqueName: \"kubernetes.io/projected/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-kube-api-access-w9wwb\") pod \"e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:47.994990 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:47.994879 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:48.131646 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:48.131617 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg"] Jul 31 14:44:48.132787 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:44:48.132755 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e78dbc_5f56_4499_8ef8_ded2f1489a5d.slice/crio-fc256a66e2ae8ea3208dfae0315640a14e06f91cfa6c5ca634a8fd5ccb7311bd WatchSource:0}: Error finding container fc256a66e2ae8ea3208dfae0315640a14e06f91cfa6c5ca634a8fd5ccb7311bd: Status 404 returned error can't find the container with id fc256a66e2ae8ea3208dfae0315640a14e06f91cfa6c5ca634a8fd5ccb7311bd Jul 31 14:44:49.084849 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:49.084812 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" event={"ID":"00e78dbc-5f56-4499-8ef8-ded2f1489a5d","Type":"ContainerStarted","Data":"ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5"} Jul 31 14:44:49.085260 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:49.084863 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" event={"ID":"00e78dbc-5f56-4499-8ef8-ded2f1489a5d","Type":"ContainerStarted","Data":"fc256a66e2ae8ea3208dfae0315640a14e06f91cfa6c5ca634a8fd5ccb7311bd"} Jul 31 14:44:49.102918 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:49.102865 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" podStartSLOduration=2.102850953 podStartE2EDuration="2.102850953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:44:49.100172211 +0000 UTC m=+1954.608551279" watchObservedRunningTime="2026-07-31 14:44:49.102850953 +0000 UTC m=+1954.611230019" Jul 31 14:44:57.995347 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:57.995308 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:57.995839 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:57.995357 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:58.007766 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:58.007739 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:44:58.146431 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:44:58.146401 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:45:14.999677 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:14.999592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:45:16.548019 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:16.547987 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg"] Jul 31 14:45:16.564795 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:16.564748 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:45:31.568601 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.568532 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" podUID="00e78dbc-5f56-4499-8ef8-ded2f1489a5d" containerName="main" containerID="cri-o://ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5" gracePeriod=45 Jul 31 14:45:31.811847 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.811819 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:45:31.903320 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903283 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-home\") pod \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " Jul 31 14:45:31.903524 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903334 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tmp-dir\") pod \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " Jul 31 14:45:31.903524 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903362 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tls-certs\") pod \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " Jul 31 14:45:31.903524 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903393 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-model-cache\") pod \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " Jul 31 14:45:31.903524 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903439 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-dshm\") pod \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " Jul 31 14:45:31.903524 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903474 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9wwb\" (UniqueName: \"kubernetes.io/projected/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-kube-api-access-w9wwb\") pod \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\" (UID: \"00e78dbc-5f56-4499-8ef8-ded2f1489a5d\") " Jul 31 14:45:31.903797 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903627 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00e78dbc-5f56-4499-8ef8-ded2f1489a5d" (UID: "00e78dbc-5f56-4499-8ef8-ded2f1489a5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:45:31.903797 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903644 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-home" (OuterVolumeSpecName: "home") pod "00e78dbc-5f56-4499-8ef8-ded2f1489a5d" (UID: "00e78dbc-5f56-4499-8ef8-ded2f1489a5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:45:31.903797 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903756 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "00e78dbc-5f56-4499-8ef8-ded2f1489a5d" (UID: "00e78dbc-5f56-4499-8ef8-ded2f1489a5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:45:31.903940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903846 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-model-cache\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:45:31.903940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903862 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-home\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:45:31.903940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.903870 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tmp-dir\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:45:31.905890 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.905868 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00e78dbc-5f56-4499-8ef8-ded2f1489a5d" (UID: "00e78dbc-5f56-4499-8ef8-ded2f1489a5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:45:31.905890 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.905878 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-dshm" (OuterVolumeSpecName: "dshm") pod "00e78dbc-5f56-4499-8ef8-ded2f1489a5d" (UID: "00e78dbc-5f56-4499-8ef8-ded2f1489a5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 14:45:31.906015 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:31.905895 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-kube-api-access-w9wwb" (OuterVolumeSpecName: "kube-api-access-w9wwb") pod "00e78dbc-5f56-4499-8ef8-ded2f1489a5d" (UID: "00e78dbc-5f56-4499-8ef8-ded2f1489a5d"). InnerVolumeSpecName "kube-api-access-w9wwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:45:32.004841 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.004798 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-dshm\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:45:32.004841 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.004833 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9wwb\" (UniqueName: \"kubernetes.io/projected/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-kube-api-access-w9wwb\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:45:32.004841 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.004844 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e78dbc-5f56-4499-8ef8-ded2f1489a5d-tls-certs\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:45:32.331139 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.331042 2558 generic.go:356] "Generic (PLEG): container finished" podID="00e78dbc-5f56-4499-8ef8-ded2f1489a5d" containerID="ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5" exitCode=0 Jul 31 14:45:32.331139 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.331101 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" Jul 31 14:45:32.331139 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.331119 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" event={"ID":"00e78dbc-5f56-4499-8ef8-ded2f1489a5d","Type":"ContainerDied","Data":"ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5"} Jul 31 14:45:32.331388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.331169 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg" event={"ID":"00e78dbc-5f56-4499-8ef8-ded2f1489a5d","Type":"ContainerDied","Data":"fc256a66e2ae8ea3208dfae0315640a14e06f91cfa6c5ca634a8fd5ccb7311bd"} Jul 31 14:45:32.331388 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.331186 2558 scope.go:117] "RemoveContainer" containerID="ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5" Jul 31 14:45:32.340589 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.340570 2558 scope.go:117] "RemoveContainer" containerID="ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5" Jul 31 14:45:32.340859 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:45:32.340839 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5\": container with ID starting with ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5 not found: ID does not exist" containerID="ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5" Jul 31 14:45:32.340944 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.340866 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5"} err="failed to get container status \"ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5\": rpc error: code = NotFound desc = could not find container \"ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5\": container with ID starting with ad5d89e1f0d46a87c4f8a04b6b5290cfb33c1046455273ef6edc07c3b6a835f5 not found: ID does not exist" Jul 31 14:45:32.353103 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.353056 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg"] Jul 31 14:45:32.355868 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.355849 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6b2a4720-kserve-bcc978887-jn6hg"] Jul 31 14:45:32.983597 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:45:32.983554 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e78dbc-5f56-4499-8ef8-ded2f1489a5d" path="/var/lib/kubelet/pods/00e78dbc-5f56-4499-8ef8-ded2f1489a5d/volumes" Jul 31 14:48:34.825975 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:48:34.825872 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:00.696964 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:00.696913 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:03.946144 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:03.946105 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:04.045765 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:04.045722 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:04.437137 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:04.437104 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:10.285382 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:10.285349 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:10.360675 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:10.360646 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:10.556803 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:10.556722 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:10.922111 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:10.922076 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:11.132863 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:11.132828 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:13.612327 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:13.612280 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:13.700701 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:13.700662 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:14.678637 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.678601 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89"] Jul 31 14:49:14.679190 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.679174 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e78dbc-5f56-4499-8ef8-ded2f1489a5d" containerName="main" Jul 31 14:49:14.679236 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.679193 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e78dbc-5f56-4499-8ef8-ded2f1489a5d" containerName="main" Jul 31 14:49:14.679318 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.679307 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e78dbc-5f56-4499-8ef8-ded2f1489a5d" containerName="main" Jul 31 14:49:14.682689 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.682672 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.685461 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.685441 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3ea37d42-serving-cert\"" Jul 31 14:49:14.692772 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.692747 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89"] Jul 31 14:49:14.707536 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.707515 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:14.842688 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.842654 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zvm\" (UniqueName: \"kubernetes.io/projected/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-kube-api-access-k4zvm\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.842858 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.842697 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-maas-api-tls\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.842858 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.842718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-openshift-service-ca\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.943881 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.943774 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zvm\" (UniqueName: \"kubernetes.io/projected/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-kube-api-access-k4zvm\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.943881 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.943836 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-maas-api-tls\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.943881 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.943867 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-openshift-service-ca\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.944530 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.944504 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-openshift-service-ca\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.946493 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.946473 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3ea37d42-serving-cert\"" Jul 31 14:49:14.955593 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.955563 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zvm\" (UniqueName: \"kubernetes.io/projected/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-kube-api-access-k4zvm\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.956573 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.956554 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-maas-api-tls\") pod \"maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:14.990150 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.990123 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:14.995125 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:14.995112 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:15.130170 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:15.130145 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89"] Jul 31 14:49:15.131897 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:49:15.131866 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e7e08d_ee44_4a4e_be3d_5ee1cfdd9d3a.slice/crio-554feb87d1f564e686815115782496df202f611d07fe24066be7ae286bec0e30 WatchSource:0}: Error finding container 554feb87d1f564e686815115782496df202f611d07fe24066be7ae286bec0e30: Status 404 returned error can't find the container with id 554feb87d1f564e686815115782496df202f611d07fe24066be7ae286bec0e30 Jul 31 14:49:15.133564 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:15.133546 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:49:15.194531 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:15.194437 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:15.295774 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:15.295717 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:15.517895 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:15.517789 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" event={"ID":"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a","Type":"ContainerStarted","Data":"554feb87d1f564e686815115782496df202f611d07fe24066be7ae286bec0e30"} Jul 31 14:49:16.530212 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:16.530178 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" event={"ID":"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a","Type":"ContainerStarted","Data":"96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba"} Jul 31 14:49:16.530699 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:16.530266 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:16.548292 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:16.548237 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" podStartSLOduration=2.104886833 podStartE2EDuration="2.548221707s" totalImagesPullingTime="443.334874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:16.546394312 +0000 UTC m=+2222.054773369" watchObservedRunningTime="2026-07-31 14:49:16.548221707 +0000 UTC m=+2222.056600773" Jul 31 14:49:22.540880 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:22.540851 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:46.309993 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:46.309959 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:55.162749 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.162707 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89"] Jul 31 14:49:55.163283 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.162999 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" podUID="76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" containerName="maas-api" containerID="cri-o://96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba" gracePeriod=30 Jul 31 14:49:55.184132 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.184108 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:55.248721 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.248676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:55.313431 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.313398 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:55.375313 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.375281 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:55.417024 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.416963 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:55.499498 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.499457 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:55.506542 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.506522 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-openshift-service-ca\") pod \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " Jul 31 14:49:55.506630 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.506556 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4zvm\" (UniqueName: \"kubernetes.io/projected/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-kube-api-access-k4zvm\") pod \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " Jul 31 14:49:55.506630 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.506623 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-maas-api-tls\") pod \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\" (UID: \"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a\") " Jul 31 14:49:55.507006 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.506926 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" (UID: "76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:49:55.508774 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.508750 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-kube-api-access-k4zvm" (OuterVolumeSpecName: "kube-api-access-k4zvm") pod "76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" (UID: "76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a"). InnerVolumeSpecName "kube-api-access-k4zvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:49:55.508863 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.508846 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" (UID: "76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:49:55.608143 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.608113 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:49:55.608143 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.608141 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:49:55.608338 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.608151 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4zvm\" (UniqueName: \"kubernetes.io/projected/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a-kube-api-access-k4zvm\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:49:55.745782 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.745684 2558 generic.go:356] "Generic (PLEG): container finished" podID="76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" containerID="96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba" exitCode=0 Jul 31 14:49:55.745782 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.745738 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" Jul 31 14:49:55.746022 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.745779 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" event={"ID":"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a","Type":"ContainerDied","Data":"96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba"} Jul 31 14:49:55.746022 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.745826 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89" event={"ID":"76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a","Type":"ContainerDied","Data":"554feb87d1f564e686815115782496df202f611d07fe24066be7ae286bec0e30"} Jul 31 14:49:55.746022 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.745846 2558 scope.go:117] "RemoveContainer" containerID="96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba" Jul 31 14:49:55.754211 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.754186 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:55.755464 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.755444 2558 scope.go:117] "RemoveContainer" containerID="96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba" Jul 31 14:49:55.755712 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:49:55.755693 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba\": container with ID starting with 96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba not found: ID does not exist" containerID="96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba" Jul 31 14:49:55.755785 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.755723 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba"} err="failed to get container status \"96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba\": rpc error: code = NotFound desc = could not find container \"96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba\": container with ID starting with 96ef7e6bd457bc07e3174ee6ba7c7bdb5c3f125895f4b951241c074b4a221eba not found: ID does not exist" Jul 31 14:49:55.767622 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.767603 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89"] Jul 31 14:49:55.771676 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:55.771656 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ea37d42-646bbd79f-4gw89"] Jul 31 14:49:56.020292 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:56.020204 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:56.118033 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:56.118001 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:56.260598 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:56.260556 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:56.667957 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:56.667926 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:56.982384 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:56.982287 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:49:56.983225 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:49:56.983199 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" path="/var/lib/kubelet/pods/76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a/volumes" Jul 31 14:50:02.731188 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.731149 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:02.805322 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.805291 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq"] Jul 31 14:50:02.805667 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.805654 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" containerName="maas-api" Jul 31 14:50:02.805667 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.805667 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" containerName="maas-api" Jul 31 14:50:02.805757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.805732 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e7e08d-ee44-4a4e-be3d-5ee1cfdd9d3a" containerName="maas-api" Jul 31 14:50:02.810317 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.810291 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:02.812800 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.812780 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6abebbcb-serving-cert\"" Jul 31 14:50:02.820148 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.820123 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq"] Jul 31 14:50:02.836461 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.836433 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:02.977849 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.977803 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcz9r\" (UniqueName: \"kubernetes.io/projected/cb661268-acea-40c2-b251-888bf3343fec-kube-api-access-hcz9r\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:02.978079 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.977865 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb661268-acea-40c2-b251-888bf3343fec-maas-api-tls\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:02.978079 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:02.977900 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb661268-acea-40c2-b251-888bf3343fec-openshift-service-ca\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.078616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.078521 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb661268-acea-40c2-b251-888bf3343fec-maas-api-tls\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.078616 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.078563 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb661268-acea-40c2-b251-888bf3343fec-openshift-service-ca\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.078856 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.078684 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz9r\" (UniqueName: \"kubernetes.io/projected/cb661268-acea-40c2-b251-888bf3343fec-kube-api-access-hcz9r\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.079363 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.079337 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb661268-acea-40c2-b251-888bf3343fec-openshift-service-ca\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.081071 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.081050 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb661268-acea-40c2-b251-888bf3343fec-maas-api-tls\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.087595 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.087572 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz9r\" (UniqueName: \"kubernetes.io/projected/cb661268-acea-40c2-b251-888bf3343fec-kube-api-access-hcz9r\") pod \"maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.094048 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.094024 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:03.123163 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.123134 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.264539 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.264507 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq"] Jul 31 14:50:03.266169 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:50:03.266140 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb661268_acea_40c2_b251_888bf3343fec.slice/crio-d718be4fea6c30813cd6e23bab0c2085260c3bd848b3cccaa63cafe1119989d3 WatchSource:0}: Error finding container d718be4fea6c30813cd6e23bab0c2085260c3bd848b3cccaa63cafe1119989d3: Status 404 returned error can't find the container with id d718be4fea6c30813cd6e23bab0c2085260c3bd848b3cccaa63cafe1119989d3 Jul 31 14:50:03.791467 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.791430 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" event={"ID":"cb661268-acea-40c2-b251-888bf3343fec","Type":"ContainerStarted","Data":"852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375"} Jul 31 14:50:03.791836 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.791490 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" event={"ID":"cb661268-acea-40c2-b251-888bf3343fec","Type":"ContainerStarted","Data":"d718be4fea6c30813cd6e23bab0c2085260c3bd848b3cccaa63cafe1119989d3"} Jul 31 14:50:03.791836 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.791516 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:03.808561 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:03.808506 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" podStartSLOduration=1.469610169 podStartE2EDuration="1.808489856s" totalImagesPullingTime="338.879687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:03.807524449 +0000 UTC m=+2269.315903539" watchObservedRunningTime="2026-07-31 14:50:03.808489856 +0000 UTC m=+2269.316868922" Jul 31 14:50:04.137625 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:04.137596 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:04.193433 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:04.193406 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:09.808530 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:09.808503 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:34.063799 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.063762 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq"] Jul 31 14:50:34.064743 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.064024 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" podUID="cb661268-acea-40c2-b251-888bf3343fec" containerName="maas-api" containerID="cri-o://852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375" gracePeriod=30 Jul 31 14:50:34.093706 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.093679 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:34.210341 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.210301 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:34.270851 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.270819 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:34.326059 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.325994 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:34.345494 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.345464 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:34.450354 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.450320 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb661268-acea-40c2-b251-888bf3343fec-openshift-service-ca\") pod \"cb661268-acea-40c2-b251-888bf3343fec\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " Jul 31 14:50:34.450537 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.450505 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb661268-acea-40c2-b251-888bf3343fec-maas-api-tls\") pod \"cb661268-acea-40c2-b251-888bf3343fec\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " Jul 31 14:50:34.450598 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.450559 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcz9r\" (UniqueName: \"kubernetes.io/projected/cb661268-acea-40c2-b251-888bf3343fec-kube-api-access-hcz9r\") pod \"cb661268-acea-40c2-b251-888bf3343fec\" (UID: \"cb661268-acea-40c2-b251-888bf3343fec\") " Jul 31 14:50:34.450684 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.450651 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb661268-acea-40c2-b251-888bf3343fec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb661268-acea-40c2-b251-888bf3343fec" (UID: "cb661268-acea-40c2-b251-888bf3343fec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:50:34.450950 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.450898 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb661268-acea-40c2-b251-888bf3343fec-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:50:34.453438 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.453401 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb661268-acea-40c2-b251-888bf3343fec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb661268-acea-40c2-b251-888bf3343fec" (UID: "cb661268-acea-40c2-b251-888bf3343fec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:50:34.453542 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.453490 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb661268-acea-40c2-b251-888bf3343fec-kube-api-access-hcz9r" (OuterVolumeSpecName: "kube-api-access-hcz9r") pod "cb661268-acea-40c2-b251-888bf3343fec" (UID: "cb661268-acea-40c2-b251-888bf3343fec"). InnerVolumeSpecName "kube-api-access-hcz9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:50:34.457375 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.457349 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:34.552192 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.552161 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb661268-acea-40c2-b251-888bf3343fec-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:50:34.552192 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.552187 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcz9r\" (UniqueName: \"kubernetes.io/projected/cb661268-acea-40c2-b251-888bf3343fec-kube-api-access-hcz9r\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:50:34.725076 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.725041 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:34.957419 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.957378 2558 generic.go:356] "Generic (PLEG): container finished" podID="cb661268-acea-40c2-b251-888bf3343fec" containerID="852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375" exitCode=0 Jul 31 14:50:34.957591 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.957432 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" Jul 31 14:50:34.957591 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.957466 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" event={"ID":"cb661268-acea-40c2-b251-888bf3343fec","Type":"ContainerDied","Data":"852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375"} Jul 31 14:50:34.957591 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.957525 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq" event={"ID":"cb661268-acea-40c2-b251-888bf3343fec","Type":"ContainerDied","Data":"d718be4fea6c30813cd6e23bab0c2085260c3bd848b3cccaa63cafe1119989d3"} Jul 31 14:50:34.957591 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.957547 2558 scope.go:117] "RemoveContainer" containerID="852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375" Jul 31 14:50:34.967477 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.967446 2558 scope.go:117] "RemoveContainer" containerID="852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375" Jul 31 14:50:34.967751 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:50:34.967721 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375\": container with ID starting with 852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375 not found: ID does not exist" containerID="852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375" Jul 31 14:50:34.967839 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.967756 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375"} err="failed to get container status \"852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375\": rpc error: code = NotFound desc = could not find container \"852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375\": container with ID starting with 852b7e98bff2c0fec893798e2faec8408645d1e3ed5065f6273331720b564375 not found: ID does not exist" Jul 31 14:50:34.968201 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.968184 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:34.990509 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.990446 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq"] Jul 31 14:50:34.990509 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:34.990479 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6abebbcb-85fbc77664-r8blq"] Jul 31 14:50:35.274547 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:35.274466 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:35.422086 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:35.422051 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:36.982859 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:36.982827 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb661268-acea-40c2-b251-888bf3343fec" path="/var/lib/kubelet/pods/cb661268-acea-40c2-b251-888bf3343fec/volumes" Jul 31 14:50:37.696051 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:37.696011 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:37.794608 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:37.794576 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:38.792839 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.792789 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5"] Jul 31 14:50:38.793423 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.793402 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb661268-acea-40c2-b251-888bf3343fec" containerName="maas-api" Jul 31 14:50:38.793508 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.793425 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb661268-acea-40c2-b251-888bf3343fec" containerName="maas-api" Jul 31 14:50:38.793576 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.793531 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb661268-acea-40c2-b251-888bf3343fec" containerName="maas-api" Jul 31 14:50:38.797954 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.797932 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:38.800897 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.800873 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6c458921-serving-cert\"" Jul 31 14:50:38.808250 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.808227 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5"] Jul 31 14:50:38.827719 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.827686 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:38.894019 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.893985 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f888f94f-4836-49a7-a0aa-120a729a9d6b-openshift-service-ca\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:38.894192 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.894117 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4ft\" (UniqueName: \"kubernetes.io/projected/f888f94f-4836-49a7-a0aa-120a729a9d6b-kube-api-access-xw4ft\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:38.894192 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.894164 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:38.995429 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.995385 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4ft\" (UniqueName: \"kubernetes.io/projected/f888f94f-4836-49a7-a0aa-120a729a9d6b-kube-api-access-xw4ft\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:38.995429 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.995433 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:38.995714 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.995477 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f888f94f-4836-49a7-a0aa-120a729a9d6b-openshift-service-ca\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:38.995714 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:50:38.995602 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-serving-cert: secret "maas-api-e2e-ait-6c458921-serving-cert" not found Jul 31 14:50:38.995714 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:50:38.995669 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls podName:f888f94f-4836-49a7-a0aa-120a729a9d6b nodeName:}" failed. No retries permitted until 2026-07-31 14:50:39.495652765 +0000 UTC m=+2305.004031809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls") pod "maas-api-e2e-ait-6c458921-85db979c7b-ttch5" (UID: "f888f94f-4836-49a7-a0aa-120a729a9d6b") : secret "maas-api-e2e-ait-6c458921-serving-cert" not found Jul 31 14:50:38.996242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:38.996217 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f888f94f-4836-49a7-a0aa-120a729a9d6b-openshift-service-ca\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:39.004732 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.004692 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4ft\" (UniqueName: \"kubernetes.io/projected/f888f94f-4836-49a7-a0aa-120a729a9d6b-kube-api-access-xw4ft\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:39.290760 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.290726 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:39.390021 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.389990 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:39.391589 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.391567 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj"] Jul 31 14:50:39.395259 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.395238 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:39.398584 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.398555 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6c458921-dockercfg-xjxcb\"" Jul 31 14:50:39.398837 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.398818 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6c458921\"" Jul 31 14:50:39.412549 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.412507 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj"] Jul 31 14:50:39.500100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.500032 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:39.500305 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.500103 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854fb5f4-9336-4411-b0ff-dbec2369df96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj\" (UID: \"854fb5f4-9336-4411-b0ff-dbec2369df96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:39.500305 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.500136 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxt6n\" (UniqueName: \"kubernetes.io/projected/854fb5f4-9336-4411-b0ff-dbec2369df96-kube-api-access-sxt6n\") pod \"payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj\" (UID: \"854fb5f4-9336-4411-b0ff-dbec2369df96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:39.500305 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:50:39.500227 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-serving-cert: secret "maas-api-e2e-ait-6c458921-serving-cert" not found Jul 31 14:50:39.500305 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:50:39.500303 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls podName:f888f94f-4836-49a7-a0aa-120a729a9d6b nodeName:}" failed. No retries permitted until 2026-07-31 14:50:40.500283658 +0000 UTC m=+2306.008662715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls") pod "maas-api-e2e-ait-6c458921-85db979c7b-ttch5" (UID: "f888f94f-4836-49a7-a0aa-120a729a9d6b") : secret "maas-api-e2e-ait-6c458921-serving-cert" not found Jul 31 14:50:39.600899 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.600806 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854fb5f4-9336-4411-b0ff-dbec2369df96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj\" (UID: \"854fb5f4-9336-4411-b0ff-dbec2369df96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:39.600899 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.600844 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxt6n\" (UniqueName: \"kubernetes.io/projected/854fb5f4-9336-4411-b0ff-dbec2369df96-kube-api-access-sxt6n\") pod \"payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj\" (UID: \"854fb5f4-9336-4411-b0ff-dbec2369df96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:39.601642 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.601618 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854fb5f4-9336-4411-b0ff-dbec2369df96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj\" (UID: \"854fb5f4-9336-4411-b0ff-dbec2369df96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:39.612124 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.612098 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxt6n\" (UniqueName: \"kubernetes.io/projected/854fb5f4-9336-4411-b0ff-dbec2369df96-kube-api-access-sxt6n\") pod \"payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj\" (UID: \"854fb5f4-9336-4411-b0ff-dbec2369df96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:39.706450 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.706417 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:39.857983 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.853992 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj"] Jul 31 14:50:39.987694 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.987657 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" event={"ID":"854fb5f4-9336-4411-b0ff-dbec2369df96","Type":"ContainerStarted","Data":"8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63"} Jul 31 14:50:39.987939 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.987716 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" event={"ID":"854fb5f4-9336-4411-b0ff-dbec2369df96","Type":"ContainerStarted","Data":"6606e66890593b84febeb62f97ee8c5ae8d4fc70902c62fe5aae1f0e6962153a"} Jul 31 14:50:39.987939 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:39.987759 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:40.014415 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:40.014365 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" podStartSLOduration=1.014347225 podStartE2EDuration="1.014347225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:40.011015139 +0000 UTC m=+2305.519394205" watchObservedRunningTime="2026-07-31 14:50:40.014347225 +0000 UTC m=+2305.522726290" Jul 31 14:50:40.088056 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:40.088025 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:50:40.510643 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:40.510599 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:40.513127 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:40.513103 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls\") pod \"maas-api-e2e-ait-6c458921-85db979c7b-ttch5\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:40.611344 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:40.611299 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:40.746232 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:40.746201 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5"] Jul 31 14:50:40.747746 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:50:40.747715 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf888f94f_4836_49a7_a0aa_120a729a9d6b.slice/crio-447572a2f7d370450f144f66b29d92eef56ca0bd874d799f2e7201adfac10961 WatchSource:0}: Error finding container 447572a2f7d370450f144f66b29d92eef56ca0bd874d799f2e7201adfac10961: Status 404 returned error can't find the container with id 447572a2f7d370450f144f66b29d92eef56ca0bd874d799f2e7201adfac10961 Jul 31 14:50:40.994850 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:40.994815 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" event={"ID":"f888f94f-4836-49a7-a0aa-120a729a9d6b","Type":"ContainerStarted","Data":"447572a2f7d370450f144f66b29d92eef56ca0bd874d799f2e7201adfac10961"} Jul 31 14:50:42.001953 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:42.001917 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" event={"ID":"f888f94f-4836-49a7-a0aa-120a729a9d6b","Type":"ContainerStarted","Data":"e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1"} Jul 31 14:50:42.002334 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:42.002093 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:42.020837 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:42.020794 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" podStartSLOduration=3.616394138 podStartE2EDuration="4.020780852s" totalImagesPullingTime="404.386714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:42.018775614 +0000 UTC m=+2307.527154679" watchObservedRunningTime="2026-07-31 14:50:42.020780852 +0000 UTC m=+2307.529159946" Jul 31 14:50:48.012530 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:48.012499 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:50:50.997369 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:50.997341 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:50:56.142707 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:50:56.142666 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:01.802339 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:01.802292 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:09.266105 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.266023 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5"] Jul 31 14:51:09.266546 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.266279 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" podUID="f888f94f-4836-49a7-a0aa-120a729a9d6b" containerName="maas-api" containerID="cri-o://e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1" gracePeriod=30 Jul 31 14:51:09.289550 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.289515 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:09.341334 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.341297 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj"] Jul 31 14:51:09.341618 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.341564 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" podUID="854fb5f4-9336-4411-b0ff-dbec2369df96" containerName="payload-pre-processing" containerID="cri-o://8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63" gracePeriod=30 Jul 31 14:51:09.425787 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.425745 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:09.532612 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.532499 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:09.603375 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.603339 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:09.608320 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.608296 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:51:09.623353 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.623331 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:51:09.679618 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.679584 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw4ft\" (UniqueName: \"kubernetes.io/projected/f888f94f-4836-49a7-a0aa-120a729a9d6b-kube-api-access-xw4ft\") pod \"f888f94f-4836-49a7-a0aa-120a729a9d6b\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " Jul 31 14:51:09.679618 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.679619 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854fb5f4-9336-4411-b0ff-dbec2369df96-plugins-config-volume\") pod \"854fb5f4-9336-4411-b0ff-dbec2369df96\" (UID: \"854fb5f4-9336-4411-b0ff-dbec2369df96\") " Jul 31 14:51:09.679818 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.679679 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxt6n\" (UniqueName: \"kubernetes.io/projected/854fb5f4-9336-4411-b0ff-dbec2369df96-kube-api-access-sxt6n\") pod \"854fb5f4-9336-4411-b0ff-dbec2369df96\" (UID: \"854fb5f4-9336-4411-b0ff-dbec2369df96\") " Jul 31 14:51:09.679818 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.679718 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f888f94f-4836-49a7-a0aa-120a729a9d6b-openshift-service-ca\") pod \"f888f94f-4836-49a7-a0aa-120a729a9d6b\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " Jul 31 14:51:09.679818 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.679756 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls\") pod \"f888f94f-4836-49a7-a0aa-120a729a9d6b\" (UID: \"f888f94f-4836-49a7-a0aa-120a729a9d6b\") " Jul 31 14:51:09.680112 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.680083 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f888f94f-4836-49a7-a0aa-120a729a9d6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f888f94f-4836-49a7-a0aa-120a729a9d6b" (UID: "f888f94f-4836-49a7-a0aa-120a729a9d6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:09.680112 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.680085 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854fb5f4-9336-4411-b0ff-dbec2369df96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "854fb5f4-9336-4411-b0ff-dbec2369df96" (UID: "854fb5f4-9336-4411-b0ff-dbec2369df96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:09.681861 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.681831 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f888f94f-4836-49a7-a0aa-120a729a9d6b" (UID: "f888f94f-4836-49a7-a0aa-120a729a9d6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:51:09.681984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.681895 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f888f94f-4836-49a7-a0aa-120a729a9d6b-kube-api-access-xw4ft" (OuterVolumeSpecName: "kube-api-access-xw4ft") pod "f888f94f-4836-49a7-a0aa-120a729a9d6b" (UID: "f888f94f-4836-49a7-a0aa-120a729a9d6b"). InnerVolumeSpecName "kube-api-access-xw4ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:09.681984 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.681933 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854fb5f4-9336-4411-b0ff-dbec2369df96-kube-api-access-sxt6n" (OuterVolumeSpecName: "kube-api-access-sxt6n") pod "854fb5f4-9336-4411-b0ff-dbec2369df96" (UID: "854fb5f4-9336-4411-b0ff-dbec2369df96"). InnerVolumeSpecName "kube-api-access-sxt6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:09.780688 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.780658 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxt6n\" (UniqueName: \"kubernetes.io/projected/854fb5f4-9336-4411-b0ff-dbec2369df96-kube-api-access-sxt6n\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:09.780688 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.780685 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f888f94f-4836-49a7-a0aa-120a729a9d6b-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:09.780688 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.780696 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f888f94f-4836-49a7-a0aa-120a729a9d6b-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:09.780920 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.780706 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw4ft\" (UniqueName: \"kubernetes.io/projected/f888f94f-4836-49a7-a0aa-120a729a9d6b-kube-api-access-xw4ft\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:09.780920 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:09.780714 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854fb5f4-9336-4411-b0ff-dbec2369df96-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:10.156257 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.156225 2558 generic.go:356] "Generic (PLEG): container finished" podID="f888f94f-4836-49a7-a0aa-120a729a9d6b" containerID="e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1" exitCode=0 Jul 31 14:51:10.156435 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.156271 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" Jul 31 14:51:10.156435 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.156316 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" event={"ID":"f888f94f-4836-49a7-a0aa-120a729a9d6b","Type":"ContainerDied","Data":"e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1"} Jul 31 14:51:10.156435 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.156369 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5" event={"ID":"f888f94f-4836-49a7-a0aa-120a729a9d6b","Type":"ContainerDied","Data":"447572a2f7d370450f144f66b29d92eef56ca0bd874d799f2e7201adfac10961"} Jul 31 14:51:10.156435 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.156395 2558 scope.go:117] "RemoveContainer" containerID="e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1" Jul 31 14:51:10.157808 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.157777 2558 generic.go:356] "Generic (PLEG): container finished" podID="854fb5f4-9336-4411-b0ff-dbec2369df96" containerID="8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63" exitCode=0 Jul 31 14:51:10.157952 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.157822 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" Jul 31 14:51:10.157952 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.157830 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" event={"ID":"854fb5f4-9336-4411-b0ff-dbec2369df96","Type":"ContainerDied","Data":"8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63"} Jul 31 14:51:10.157952 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.157871 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj" event={"ID":"854fb5f4-9336-4411-b0ff-dbec2369df96","Type":"ContainerDied","Data":"6606e66890593b84febeb62f97ee8c5ae8d4fc70902c62fe5aae1f0e6962153a"} Jul 31 14:51:10.166762 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.166744 2558 scope.go:117] "RemoveContainer" containerID="e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1" Jul 31 14:51:10.167059 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:51:10.167035 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1\": container with ID starting with e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1 not found: ID does not exist" containerID="e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1" Jul 31 14:51:10.167168 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.167070 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1"} err="failed to get container status \"e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1\": rpc error: code = NotFound desc = could not find container \"e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1\": container with ID starting with e401ecf98a8a7de886bdb88bbcffd4bf0c5360bc44ef175a9c54040a38a9efd1 not found: ID does not exist" Jul 31 14:51:10.167168 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.167092 2558 scope.go:117] "RemoveContainer" containerID="8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63" Jul 31 14:51:10.175440 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.175421 2558 scope.go:117] "RemoveContainer" containerID="8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63" Jul 31 14:51:10.175701 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:51:10.175681 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63\": container with ID starting with 8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63 not found: ID does not exist" containerID="8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63" Jul 31 14:51:10.175767 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.175708 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63"} err="failed to get container status \"8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63\": rpc error: code = NotFound desc = could not find container \"8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63\": container with ID starting with 8dea58f62a01e06e456b36db33284b16ae6439a733c2033be63adb6012378c63 not found: ID does not exist" Jul 31 14:51:10.179893 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.179872 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5"] Jul 31 14:51:10.185315 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.185295 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6c458921-85db979c7b-ttch5"] Jul 31 14:51:10.196245 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.196226 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj"] Jul 31 14:51:10.198323 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.198305 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6c458921-866657b956-4k9vj"] Jul 31 14:51:10.804691 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.804658 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:10.984021 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.983988 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="854fb5f4-9336-4411-b0ff-dbec2369df96" path="/var/lib/kubelet/pods/854fb5f4-9336-4411-b0ff-dbec2369df96/volumes" Jul 31 14:51:10.984405 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:10.984391 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f888f94f-4836-49a7-a0aa-120a729a9d6b" path="/var/lib/kubelet/pods/f888f94f-4836-49a7-a0aa-120a729a9d6b/volumes" Jul 31 14:51:11.106491 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:11.106382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:11.291066 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:11.291022 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:11.623040 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:11.623002 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:11.965239 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:11.965122 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:13.912554 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:13.912521 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:14.017212 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:14.017175 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:15.004674 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.004636 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt"] Jul 31 14:51:15.005156 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.005138 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f888f94f-4836-49a7-a0aa-120a729a9d6b" containerName="maas-api" Jul 31 14:51:15.005204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.005158 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f888f94f-4836-49a7-a0aa-120a729a9d6b" containerName="maas-api" Jul 31 14:51:15.005204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.005175 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="854fb5f4-9336-4411-b0ff-dbec2369df96" containerName="payload-pre-processing" Jul 31 14:51:15.005204 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.005181 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="854fb5f4-9336-4411-b0ff-dbec2369df96" containerName="payload-pre-processing" Jul 31 14:51:15.005319 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.005263 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f888f94f-4836-49a7-a0aa-120a729a9d6b" containerName="maas-api" Jul 31 14:51:15.005319 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.005274 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="854fb5f4-9336-4411-b0ff-dbec2369df96" containerName="payload-pre-processing" Jul 31 14:51:15.010130 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.010109 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.013635 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.013617 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3187acc6-serving-cert\"" Jul 31 14:51:15.022860 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.022833 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt"] Jul 31 14:51:15.119331 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.119301 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:15.130500 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.130470 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c35171-3af5-4b08-83b7-ff58432a60a9-maas-api-tls\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.130635 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.130507 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c35171-3af5-4b08-83b7-ff58432a60a9-openshift-service-ca\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.130635 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.130535 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjq9\" (UniqueName: \"kubernetes.io/projected/93c35171-3af5-4b08-83b7-ff58432a60a9-kube-api-access-4hjq9\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.205530 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.205503 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:15.231895 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.231865 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c35171-3af5-4b08-83b7-ff58432a60a9-maas-api-tls\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.232047 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.231923 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c35171-3af5-4b08-83b7-ff58432a60a9-openshift-service-ca\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.232047 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.232036 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjq9\" (UniqueName: \"kubernetes.io/projected/93c35171-3af5-4b08-83b7-ff58432a60a9-kube-api-access-4hjq9\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.232641 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.232615 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c35171-3af5-4b08-83b7-ff58432a60a9-openshift-service-ca\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.234285 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.234266 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c35171-3af5-4b08-83b7-ff58432a60a9-maas-api-tls\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.240828 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.240806 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjq9\" (UniqueName: \"kubernetes.io/projected/93c35171-3af5-4b08-83b7-ff58432a60a9-kube-api-access-4hjq9\") pod \"maas-api-e2e-derive-3187acc6-66bd47965f-4twnt\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.323352 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.323261 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:15.473558 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.473534 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt"] Jul 31 14:51:15.475418 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:51:15.475387 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c35171_3af5_4b08_83b7_ff58432a60a9.slice/crio-5bc3257a811e4c538af2b0475e774f84ba5ebe866086418e6c6c41a8f470573b WatchSource:0}: Error finding container 5bc3257a811e4c538af2b0475e774f84ba5ebe866086418e6c6c41a8f470573b: Status 404 returned error can't find the container with id 5bc3257a811e4c538af2b0475e774f84ba5ebe866086418e6c6c41a8f470573b Jul 31 14:51:15.597359 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.597240 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:15.693594 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.693561 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:15.737919 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.737875 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr"] Jul 31 14:51:15.742574 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.742552 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:15.745644 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.745615 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3187acc6-dockercfg-6v9r2\"" Jul 31 14:51:15.745784 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.745649 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3187acc6\"" Jul 31 14:51:15.750577 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.750551 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr"] Jul 31 14:51:15.836724 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.836682 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/17625c8f-acb0-4ded-8a1e-12273d62ca8d-kube-api-access-tccft\") pod \"payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr\" (UID: \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\") " pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:15.836972 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.836755 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17625c8f-acb0-4ded-8a1e-12273d62ca8d-plugins-config-volume\") pod \"payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr\" (UID: \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\") " pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:15.937884 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.937837 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/17625c8f-acb0-4ded-8a1e-12273d62ca8d-kube-api-access-tccft\") pod \"payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr\" (UID: \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\") " pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:15.938090 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.938000 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17625c8f-acb0-4ded-8a1e-12273d62ca8d-plugins-config-volume\") pod \"payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr\" (UID: \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\") " pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:15.938727 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.938702 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17625c8f-acb0-4ded-8a1e-12273d62ca8d-plugins-config-volume\") pod \"payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr\" (UID: \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\") " pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:15.946251 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:15.946222 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/17625c8f-acb0-4ded-8a1e-12273d62ca8d-kube-api-access-tccft\") pod \"payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr\" (UID: \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\") " pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:16.054581 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:16.054548 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:16.194130 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:16.193823 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" event={"ID":"93c35171-3af5-4b08-83b7-ff58432a60a9","Type":"ContainerStarted","Data":"5bc3257a811e4c538af2b0475e774f84ba5ebe866086418e6c6c41a8f470573b"} Jul 31 14:51:16.196532 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:16.196505 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr"] Jul 31 14:51:16.197450 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:51:16.197424 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17625c8f_acb0_4ded_8a1e_12273d62ca8d.slice/crio-04d2b5ad00ac4349164787ee51d6adf4429a10aa445cb49dd04afaaf7c573bce WatchSource:0}: Error finding container 04d2b5ad00ac4349164787ee51d6adf4429a10aa445cb49dd04afaaf7c573bce: Status 404 returned error can't find the container with id 04d2b5ad00ac4349164787ee51d6adf4429a10aa445cb49dd04afaaf7c573bce Jul 31 14:51:17.200977 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:17.200938 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" event={"ID":"93c35171-3af5-4b08-83b7-ff58432a60a9","Type":"ContainerStarted","Data":"0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505"} Jul 31 14:51:17.201437 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:17.201168 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:17.202559 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:17.202532 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" event={"ID":"17625c8f-acb0-4ded-8a1e-12273d62ca8d","Type":"ContainerStarted","Data":"bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5"} Jul 31 14:51:17.202660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:17.202578 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" event={"ID":"17625c8f-acb0-4ded-8a1e-12273d62ca8d","Type":"ContainerStarted","Data":"04d2b5ad00ac4349164787ee51d6adf4429a10aa445cb49dd04afaaf7c573bce"} Jul 31 14:51:17.202660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:17.202599 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:17.219220 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:17.219168 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" podStartSLOduration=2.50201405 podStartE2EDuration="3.219152321s" totalImagesPullingTime="717.138271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:17.217300136 +0000 UTC m=+2342.725679202" watchObservedRunningTime="2026-07-31 14:51:17.219152321 +0000 UTC m=+2342.727531387" Jul 31 14:51:17.235896 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:17.235850 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" podStartSLOduration=2.235836197 podStartE2EDuration="2.235836197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:17.233201214 +0000 UTC m=+2342.741580281" watchObservedRunningTime="2026-07-31 14:51:17.235836197 +0000 UTC m=+2342.744215262" Jul 31 14:51:23.213001 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:23.212961 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:28.209861 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:28.209831 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:45.494525 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.494489 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt"] Jul 31 14:51:45.495139 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.494693 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" podUID="93c35171-3af5-4b08-83b7-ff58432a60a9" containerName="maas-api" containerID="cri-o://0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505" gracePeriod=30 Jul 31 14:51:45.526839 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.526813 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:45.546508 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.546474 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr"] Jul 31 14:51:45.546756 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.546722 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" podUID="17625c8f-acb0-4ded-8a1e-12273d62ca8d" containerName="payload-processing" containerID="cri-o://bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5" gracePeriod=30 Jul 31 14:51:45.629880 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.629843 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:45.683703 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.683615 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:45.801138 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.801108 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:45.825248 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.825206 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:45.829555 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.829528 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:45.920565 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.920531 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c35171-3af5-4b08-83b7-ff58432a60a9-openshift-service-ca\") pod \"93c35171-3af5-4b08-83b7-ff58432a60a9\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " Jul 31 14:51:45.920773 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.920589 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c35171-3af5-4b08-83b7-ff58432a60a9-maas-api-tls\") pod \"93c35171-3af5-4b08-83b7-ff58432a60a9\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " Jul 31 14:51:45.920773 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.920655 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hjq9\" (UniqueName: \"kubernetes.io/projected/93c35171-3af5-4b08-83b7-ff58432a60a9-kube-api-access-4hjq9\") pod \"93c35171-3af5-4b08-83b7-ff58432a60a9\" (UID: \"93c35171-3af5-4b08-83b7-ff58432a60a9\") " Jul 31 14:51:45.921075 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.921048 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c35171-3af5-4b08-83b7-ff58432a60a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93c35171-3af5-4b08-83b7-ff58432a60a9" (UID: "93c35171-3af5-4b08-83b7-ff58432a60a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:45.922788 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.922765 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c35171-3af5-4b08-83b7-ff58432a60a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93c35171-3af5-4b08-83b7-ff58432a60a9" (UID: "93c35171-3af5-4b08-83b7-ff58432a60a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:51:45.922860 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:45.922793 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c35171-3af5-4b08-83b7-ff58432a60a9-kube-api-access-4hjq9" (OuterVolumeSpecName: "kube-api-access-4hjq9") pod "93c35171-3af5-4b08-83b7-ff58432a60a9" (UID: "93c35171-3af5-4b08-83b7-ff58432a60a9"). InnerVolumeSpecName "kube-api-access-4hjq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:46.034979 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.034173 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/17625c8f-acb0-4ded-8a1e-12273d62ca8d-kube-api-access-tccft\") pod \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\" (UID: \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\") " Jul 31 14:51:46.034979 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.034240 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17625c8f-acb0-4ded-8a1e-12273d62ca8d-plugins-config-volume\") pod \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\" (UID: \"17625c8f-acb0-4ded-8a1e-12273d62ca8d\") " Jul 31 14:51:46.034979 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.034597 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjq9\" (UniqueName: \"kubernetes.io/projected/93c35171-3af5-4b08-83b7-ff58432a60a9-kube-api-access-4hjq9\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:46.034979 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.034616 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c35171-3af5-4b08-83b7-ff58432a60a9-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:46.034979 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.034630 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c35171-3af5-4b08-83b7-ff58432a60a9-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:46.035355 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.035309 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17625c8f-acb0-4ded-8a1e-12273d62ca8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17625c8f-acb0-4ded-8a1e-12273d62ca8d" (UID: "17625c8f-acb0-4ded-8a1e-12273d62ca8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:51:46.038251 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.038221 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17625c8f-acb0-4ded-8a1e-12273d62ca8d-kube-api-access-tccft" (OuterVolumeSpecName: "kube-api-access-tccft") pod "17625c8f-acb0-4ded-8a1e-12273d62ca8d" (UID: "17625c8f-acb0-4ded-8a1e-12273d62ca8d"). InnerVolumeSpecName "kube-api-access-tccft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:51:46.047869 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.047836 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:46.135226 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.135190 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/17625c8f-acb0-4ded-8a1e-12273d62ca8d-kube-api-access-tccft\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:46.135226 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.135218 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17625c8f-acb0-4ded-8a1e-12273d62ca8d-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:51:46.363300 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.363209 2558 generic.go:356] "Generic (PLEG): container finished" podID="93c35171-3af5-4b08-83b7-ff58432a60a9" containerID="0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505" exitCode=0 Jul 31 14:51:46.363300 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.363256 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" Jul 31 14:51:46.363300 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.363264 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" event={"ID":"93c35171-3af5-4b08-83b7-ff58432a60a9","Type":"ContainerDied","Data":"0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505"} Jul 31 14:51:46.363574 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.363321 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt" event={"ID":"93c35171-3af5-4b08-83b7-ff58432a60a9","Type":"ContainerDied","Data":"5bc3257a811e4c538af2b0475e774f84ba5ebe866086418e6c6c41a8f470573b"} Jul 31 14:51:46.363574 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.363345 2558 scope.go:117] "RemoveContainer" containerID="0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505" Jul 31 14:51:46.364724 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.364702 2558 generic.go:356] "Generic (PLEG): container finished" podID="17625c8f-acb0-4ded-8a1e-12273d62ca8d" containerID="bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5" exitCode=0 Jul 31 14:51:46.364848 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.364732 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" event={"ID":"17625c8f-acb0-4ded-8a1e-12273d62ca8d","Type":"ContainerDied","Data":"bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5"} Jul 31 14:51:46.364848 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.364763 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" Jul 31 14:51:46.364848 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.364785 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr" event={"ID":"17625c8f-acb0-4ded-8a1e-12273d62ca8d","Type":"ContainerDied","Data":"04d2b5ad00ac4349164787ee51d6adf4429a10aa445cb49dd04afaaf7c573bce"} Jul 31 14:51:46.374740 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.374633 2558 scope.go:117] "RemoveContainer" containerID="0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505" Jul 31 14:51:46.375032 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:51:46.375012 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505\": container with ID starting with 0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505 not found: ID does not exist" containerID="0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505" Jul 31 14:51:46.375126 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.375043 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505"} err="failed to get container status \"0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505\": rpc error: code = NotFound desc = could not find container \"0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505\": container with ID starting with 0a9405c7224a296b7853ea8565b5beb5199ebc7bd1bf49199e5dd55e32fc2505 not found: ID does not exist" Jul 31 14:51:46.375126 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.375060 2558 scope.go:117] "RemoveContainer" containerID="bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5" Jul 31 14:51:46.383702 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.383683 2558 scope.go:117] "RemoveContainer" containerID="bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5" Jul 31 14:51:46.384202 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:51:46.384176 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5\": container with ID starting with bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5 not found: ID does not exist" containerID="bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5" Jul 31 14:51:46.384302 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.384208 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5"} err="failed to get container status \"bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5\": rpc error: code = NotFound desc = could not find container \"bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5\": container with ID starting with bf8270975cb34984b3bac338f9d798aea5f4ba0daef7dedda7a63f573f4d5cd5 not found: ID does not exist" Jul 31 14:51:46.386837 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.386818 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt"] Jul 31 14:51:46.392555 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.392534 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3187acc6-66bd47965f-4twnt"] Jul 31 14:51:46.402545 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.402522 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr"] Jul 31 14:51:46.404627 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.404610 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3187acc6-78c8c8b7d-k4dkr"] Jul 31 14:51:46.526996 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.526963 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:46.631203 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.631170 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:46.729144 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.729113 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:51:46.983833 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.983751 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17625c8f-acb0-4ded-8a1e-12273d62ca8d" path="/var/lib/kubelet/pods/17625c8f-acb0-4ded-8a1e-12273d62ca8d/volumes" Jul 31 14:51:46.984209 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:46.984193 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c35171-3af5-4b08-83b7-ff58432a60a9" path="/var/lib/kubelet/pods/93c35171-3af5-4b08-83b7-ff58432a60a9/volumes" Jul 31 14:51:58.415309 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:51:58.415279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:04.510706 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:04.510653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:04.615358 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:04.615328 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:04.715228 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:04.715187 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:04.812171 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:04.812074 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:05.259361 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:05.259329 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:20.639450 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:20.639413 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:31.257812 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:31.257777 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:37.044408 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:37.044365 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:37.100399 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:37.100358 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:37.174775 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:37.174739 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:37.272119 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:37.272084 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:53.179242 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:53.179208 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:58.922428 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:58.922386 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:58.990295 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:58.990257 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:59.066945 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:59.066890 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:52:59.166460 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:52:59.166426 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:09.685922 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:09.685877 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:20.776689 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:20.776645 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:26.767885 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:26.767851 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:26.872989 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:26.872956 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:26.969445 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:26.969407 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:27.068234 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:27.068147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:27.756596 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:27.756563 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:27.852488 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:27.852455 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:27.952273 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:27.952243 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:29.511965 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:29.511928 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:35.704198 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:35.704165 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:35.799585 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:35.799541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:36.445946 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:36.445887 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:42.670096 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:42.670058 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:42.771722 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:42.771688 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:42.908105 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:42.908071 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:43.246223 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:43.246188 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:43.392689 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:43.392649 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:46.141276 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:46.141228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:46.257260 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:46.257226 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:47.231403 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.231367 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7"] Jul 31 14:53:47.231983 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.231964 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c35171-3af5-4b08-83b7-ff58432a60a9" containerName="maas-api" Jul 31 14:53:47.232046 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.231985 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c35171-3af5-4b08-83b7-ff58432a60a9" containerName="maas-api" Jul 31 14:53:47.232046 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.232020 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17625c8f-acb0-4ded-8a1e-12273d62ca8d" containerName="payload-processing" Jul 31 14:53:47.232046 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.232029 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17625c8f-acb0-4ded-8a1e-12273d62ca8d" containerName="payload-processing" Jul 31 14:53:47.232141 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.232117 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c35171-3af5-4b08-83b7-ff58432a60a9" containerName="maas-api" Jul 31 14:53:47.232141 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.232129 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="17625c8f-acb0-4ded-8a1e-12273d62ca8d" containerName="payload-processing" Jul 31 14:53:47.235734 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.235712 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.239354 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.239332 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7f0814aa-serving-cert\"" Jul 31 14:53:47.246750 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.246726 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7"] Jul 31 14:53:47.264569 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.264539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:47.339068 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.339024 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.339269 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.339109 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2285l\" (UniqueName: \"kubernetes.io/projected/cbeb3805-6473-4011-8c5f-b249dee58248-kube-api-access-2285l\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.339269 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.339169 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbeb3805-6473-4011-8c5f-b249dee58248-openshift-service-ca\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.440054 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.440016 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2285l\" (UniqueName: \"kubernetes.io/projected/cbeb3805-6473-4011-8c5f-b249dee58248-kube-api-access-2285l\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.440241 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.440082 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbeb3805-6473-4011-8c5f-b249dee58248-openshift-service-ca\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.440241 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.440127 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.440340 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:53:47.440240 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-serving-cert: secret "maas-api-e2e-mt-7f0814aa-serving-cert" not found Jul 31 14:53:47.440340 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:53:47.440294 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls podName:cbeb3805-6473-4011-8c5f-b249dee58248 nodeName:}" failed. No retries permitted until 2026-07-31 14:53:47.940278174 +0000 UTC m=+2493.448657217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls") pod "maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" (UID: "cbeb3805-6473-4011-8c5f-b249dee58248") : secret "maas-api-e2e-mt-7f0814aa-serving-cert" not found Jul 31 14:53:47.440704 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.440685 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbeb3805-6473-4011-8c5f-b249dee58248-openshift-service-ca\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.449184 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.449162 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2285l\" (UniqueName: \"kubernetes.io/projected/cbeb3805-6473-4011-8c5f-b249dee58248-kube-api-access-2285l\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.745003 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.744969 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:47.837538 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.837505 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:53:47.945172 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.945134 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:47.947640 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:47.947620 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls\") pod \"maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:48.148480 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:48.148433 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:48.286825 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:48.286791 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7"] Jul 31 14:53:48.288482 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:53:48.288456 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbeb3805_6473_4011_8c5f_b249dee58248.slice/crio-291553b0df1a5192de62188d0a99626109e9b7cd1dc4f8c7a892bad49e8243f2 WatchSource:0}: Error finding container 291553b0df1a5192de62188d0a99626109e9b7cd1dc4f8c7a892bad49e8243f2: Status 404 returned error can't find the container with id 291553b0df1a5192de62188d0a99626109e9b7cd1dc4f8c7a892bad49e8243f2 Jul 31 14:53:49.051978 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:49.051871 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" event={"ID":"cbeb3805-6473-4011-8c5f-b249dee58248","Type":"ContainerStarted","Data":"d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b"} Jul 31 14:53:49.052142 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:49.052003 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" event={"ID":"cbeb3805-6473-4011-8c5f-b249dee58248","Type":"ContainerStarted","Data":"291553b0df1a5192de62188d0a99626109e9b7cd1dc4f8c7a892bad49e8243f2"} Jul 31 14:53:49.052142 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:49.052032 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:53:49.070141 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:49.070090 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" podStartSLOduration=1.577129459 podStartE2EDuration="2.070075723s" totalImagesPullingTime="492.946264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:53:49.068029967 +0000 UTC m=+2494.576409034" watchObservedRunningTime="2026-07-31 14:53:49.070075723 +0000 UTC m=+2494.578454798" Jul 31 14:53:55.062991 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:53:55.062961 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:54:07.583697 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:07.583612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:11.158098 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:11.158063 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:11.884007 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:11.883970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:11.984809 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:11.984775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:12.331110 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:12.331027 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:17.936618 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:17.936554 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:17.998482 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:17.998447 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:28.817289 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:28.817251 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7"] Jul 31 14:54:28.817693 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:28.817461 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" podUID="cbeb3805-6473-4011-8c5f-b249dee58248" containerName="maas-api" containerID="cri-o://d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b" gracePeriod=30 Jul 31 14:54:28.831533 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:28.831500 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:28.903138 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:28.903104 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:28.967770 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:28.967724 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:29.030343 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.030312 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:29.068586 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.068526 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:54:29.133444 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.133410 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbeb3805-6473-4011-8c5f-b249dee58248-openshift-service-ca\") pod \"cbeb3805-6473-4011-8c5f-b249dee58248\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " Jul 31 14:54:29.133623 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.133479 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls\") pod \"cbeb3805-6473-4011-8c5f-b249dee58248\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " Jul 31 14:54:29.133623 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.133547 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2285l\" (UniqueName: \"kubernetes.io/projected/cbeb3805-6473-4011-8c5f-b249dee58248-kube-api-access-2285l\") pod \"cbeb3805-6473-4011-8c5f-b249dee58248\" (UID: \"cbeb3805-6473-4011-8c5f-b249dee58248\") " Jul 31 14:54:29.133798 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.133776 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbeb3805-6473-4011-8c5f-b249dee58248-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbeb3805-6473-4011-8c5f-b249dee58248" (UID: "cbeb3805-6473-4011-8c5f-b249dee58248"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:54:29.135701 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.135670 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbeb3805-6473-4011-8c5f-b249dee58248-kube-api-access-2285l" (OuterVolumeSpecName: "kube-api-access-2285l") pod "cbeb3805-6473-4011-8c5f-b249dee58248" (UID: "cbeb3805-6473-4011-8c5f-b249dee58248"). InnerVolumeSpecName "kube-api-access-2285l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:54:29.135701 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.135694 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbeb3805-6473-4011-8c5f-b249dee58248" (UID: "cbeb3805-6473-4011-8c5f-b249dee58248"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:54:29.235047 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.235003 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2285l\" (UniqueName: \"kubernetes.io/projected/cbeb3805-6473-4011-8c5f-b249dee58248-kube-api-access-2285l\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:54:29.235047 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.235040 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbeb3805-6473-4011-8c5f-b249dee58248-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:54:29.235047 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.235050 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbeb3805-6473-4011-8c5f-b249dee58248-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:54:29.262524 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.262493 2558 generic.go:356] "Generic (PLEG): container finished" podID="cbeb3805-6473-4011-8c5f-b249dee58248" containerID="d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b" exitCode=0 Jul 31 14:54:29.262666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.262541 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" Jul 31 14:54:29.262666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.262546 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" event={"ID":"cbeb3805-6473-4011-8c5f-b249dee58248","Type":"ContainerDied","Data":"d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b"} Jul 31 14:54:29.262666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.262588 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7" event={"ID":"cbeb3805-6473-4011-8c5f-b249dee58248","Type":"ContainerDied","Data":"291553b0df1a5192de62188d0a99626109e9b7cd1dc4f8c7a892bad49e8243f2"} Jul 31 14:54:29.262666 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.262611 2558 scope.go:117] "RemoveContainer" containerID="d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b" Jul 31 14:54:29.273111 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.273091 2558 scope.go:117] "RemoveContainer" containerID="d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b" Jul 31 14:54:29.273378 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:54:29.273356 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b\": container with ID starting with d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b not found: ID does not exist" containerID="d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b" Jul 31 14:54:29.273438 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.273388 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b"} err="failed to get container status \"d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b\": rpc error: code = NotFound desc = could not find container \"d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b\": container with ID starting with d30df67ba6ba15b72b44dbefb435040170e7ad5f06a02887521bb198a0dfb81b not found: ID does not exist" Jul 31 14:54:29.285988 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.285960 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7"] Jul 31 14:54:29.289898 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.289867 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f0814aa-57d694cd6c-vm8h7"] Jul 31 14:54:29.383015 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:29.382976 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:30.983515 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:30.983475 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbeb3805-6473-4011-8c5f-b249dee58248" path="/var/lib/kubelet/pods/cbeb3805-6473-4011-8c5f-b249dee58248/volumes" Jul 31 14:54:35.169326 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:35.169291 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:35.224714 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:35.224679 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:35.438432 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:35.438354 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:35.667227 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:35.667187 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:35.804569 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:35.804486 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:35.908010 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:35.907975 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:36.019343 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:36.019308 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:41.620295 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:41.620228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:41.735105 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:41.735063 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:41.867395 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:41.867364 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:42.011182 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:42.011092 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:47.855881 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:47.855827 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:54:47.959077 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:54:47.959046 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:50.276714 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:50.276673 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:50.573064 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:50.572983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:50.740166 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:50.740126 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:53.999256 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:53.999137 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:54.106940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:54.106875 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:55.070702 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.070669 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn"] Jul 31 14:56:55.071287 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.071242 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbeb3805-6473-4011-8c5f-b249dee58248" containerName="maas-api" Jul 31 14:56:55.071391 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.071290 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbeb3805-6473-4011-8c5f-b249dee58248" containerName="maas-api" Jul 31 14:56:55.071391 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.071381 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbeb3805-6473-4011-8c5f-b249dee58248" containerName="maas-api" Jul 31 14:56:55.073533 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.073513 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.076011 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.075993 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-86b90147-serving-cert\"" Jul 31 14:56:55.085660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.085637 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn"] Jul 31 14:56:55.108096 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.108069 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:55.183123 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.183086 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9056bf3a-6019-4689-bf9d-932e77d0b505-maas-api-tls\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.183302 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.183143 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9056bf3a-6019-4689-bf9d-932e77d0b505-openshift-service-ca\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.183302 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.183198 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86b8q\" (UniqueName: \"kubernetes.io/projected/9056bf3a-6019-4689-bf9d-932e77d0b505-kube-api-access-86b8q\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.284681 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.284646 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9056bf3a-6019-4689-bf9d-932e77d0b505-maas-api-tls\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.284863 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.284707 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9056bf3a-6019-4689-bf9d-932e77d0b505-openshift-service-ca\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.284863 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.284739 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86b8q\" (UniqueName: \"kubernetes.io/projected/9056bf3a-6019-4689-bf9d-932e77d0b505-kube-api-access-86b8q\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.285337 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.285316 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9056bf3a-6019-4689-bf9d-932e77d0b505-openshift-service-ca\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.287205 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.287182 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9056bf3a-6019-4689-bf9d-932e77d0b505-maas-api-tls\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.309372 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.309342 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86b8q\" (UniqueName: \"kubernetes.io/projected/9056bf3a-6019-4689-bf9d-932e77d0b505-kube-api-access-86b8q\") pod \"maas-api-e2e-mt-86b90147-5785bf9676-jp5nn\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.345668 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.345582 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:55.387384 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.387351 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:55.526389 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.526361 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn"] Jul 31 14:56:55.528051 ip-10-0-130-33 kubenswrapper[2558]: W0731 14:56:55.528028 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9056bf3a_6019_4689_bf9d_932e77d0b505.slice/crio-235b051b58e6e2aae20eaae5c4168d33c4857038d9da1b8c82d7c3e1872a3613 WatchSource:0}: Error finding container 235b051b58e6e2aae20eaae5c4168d33c4857038d9da1b8c82d7c3e1872a3613: Status 404 returned error can't find the container with id 235b051b58e6e2aae20eaae5c4168d33c4857038d9da1b8c82d7c3e1872a3613 Jul 31 14:56:55.529433 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.529416 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 14:56:55.611865 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.611761 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:55.700590 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:55.700556 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:56:56.036084 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:56.036052 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" event={"ID":"9056bf3a-6019-4689-bf9d-932e77d0b505","Type":"ContainerStarted","Data":"235b051b58e6e2aae20eaae5c4168d33c4857038d9da1b8c82d7c3e1872a3613"} Jul 31 14:56:57.043621 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:57.043581 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" event={"ID":"9056bf3a-6019-4689-bf9d-932e77d0b505","Type":"ContainerStarted","Data":"b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b"} Jul 31 14:56:57.044123 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:57.043733 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:56:57.060795 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:56:57.060740 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" podStartSLOduration=1.5850091769999999 podStartE2EDuration="2.060721629s" totalImagesPullingTime="475.712452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:57.059316177 +0000 UTC m=+2682.567695243" watchObservedRunningTime="2026-07-31 14:56:57.060721629 +0000 UTC m=+2682.569100729" Jul 31 14:57:03.055136 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:57:03.055107 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:57:04.127748 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:57:04.127714 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:19.037664 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:57:19.037582 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:21.711430 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:57:21.711381 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:30.212808 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:57:30.212748 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:57:44.758095 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:57:44.758032 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:01.052660 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:01.052627 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:01.152279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:01.152244 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:14.436263 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.436198 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn"] Jul 31 14:58:14.436773 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.436520 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" podUID="9056bf3a-6019-4689-bf9d-932e77d0b505" containerName="maas-api" containerID="cri-o://b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b" gracePeriod=30 Jul 31 14:58:14.478100 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.478054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:14.641765 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.641696 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:14.687487 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.687433 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:58:14.697078 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.697049 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:14.802577 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.802541 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9056bf3a-6019-4689-bf9d-932e77d0b505-openshift-service-ca\") pod \"9056bf3a-6019-4689-bf9d-932e77d0b505\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " Jul 31 14:58:14.802764 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.802616 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9056bf3a-6019-4689-bf9d-932e77d0b505-maas-api-tls\") pod \"9056bf3a-6019-4689-bf9d-932e77d0b505\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " Jul 31 14:58:14.802764 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.802669 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86b8q\" (UniqueName: \"kubernetes.io/projected/9056bf3a-6019-4689-bf9d-932e77d0b505-kube-api-access-86b8q\") pod \"9056bf3a-6019-4689-bf9d-932e77d0b505\" (UID: \"9056bf3a-6019-4689-bf9d-932e77d0b505\") " Jul 31 14:58:14.803010 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.802982 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9056bf3a-6019-4689-bf9d-932e77d0b505-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9056bf3a-6019-4689-bf9d-932e77d0b505" (UID: "9056bf3a-6019-4689-bf9d-932e77d0b505"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 14:58:14.804776 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.804756 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9056bf3a-6019-4689-bf9d-932e77d0b505-kube-api-access-86b8q" (OuterVolumeSpecName: "kube-api-access-86b8q") pod "9056bf3a-6019-4689-bf9d-932e77d0b505" (UID: "9056bf3a-6019-4689-bf9d-932e77d0b505"). InnerVolumeSpecName "kube-api-access-86b8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 14:58:14.804832 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.804804 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9056bf3a-6019-4689-bf9d-932e77d0b505-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9056bf3a-6019-4689-bf9d-932e77d0b505" (UID: "9056bf3a-6019-4689-bf9d-932e77d0b505"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 14:58:14.903556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.903525 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86b8q\" (UniqueName: \"kubernetes.io/projected/9056bf3a-6019-4689-bf9d-932e77d0b505-kube-api-access-86b8q\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:58:14.903556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.903553 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9056bf3a-6019-4689-bf9d-932e77d0b505-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:58:14.903556 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:14.903563 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9056bf3a-6019-4689-bf9d-932e77d0b505-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 14:58:15.467362 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.467327 2558 generic.go:356] "Generic (PLEG): container finished" podID="9056bf3a-6019-4689-bf9d-932e77d0b505" containerID="b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b" exitCode=0 Jul 31 14:58:15.467757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.467390 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" Jul 31 14:58:15.467757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.467418 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" event={"ID":"9056bf3a-6019-4689-bf9d-932e77d0b505","Type":"ContainerDied","Data":"b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b"} Jul 31 14:58:15.467757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.467509 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn" event={"ID":"9056bf3a-6019-4689-bf9d-932e77d0b505","Type":"ContainerDied","Data":"235b051b58e6e2aae20eaae5c4168d33c4857038d9da1b8c82d7c3e1872a3613"} Jul 31 14:58:15.467757 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.467538 2558 scope.go:117] "RemoveContainer" containerID="b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b" Jul 31 14:58:15.476845 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.476817 2558 scope.go:117] "RemoveContainer" containerID="b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b" Jul 31 14:58:15.477139 ip-10-0-130-33 kubenswrapper[2558]: E0731 14:58:15.477121 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b\": container with ID starting with b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b not found: ID does not exist" containerID="b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b" Jul 31 14:58:15.477191 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.477149 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b"} err="failed to get container status \"b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b\": rpc error: code = NotFound desc = could not find container \"b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b\": container with ID starting with b7172cf58c6443d42ad421300e481a3063ca7b6e601b047fa4ed38e101afa96b not found: ID does not exist" Jul 31 14:58:15.485014 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.484987 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn"] Jul 31 14:58:15.487156 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:15.487135 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-86b90147-5785bf9676-jp5nn"] Jul 31 14:58:16.984026 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:16.983993 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9056bf3a-6019-4689-bf9d-932e77d0b505" path="/var/lib/kubelet/pods/9056bf3a-6019-4689-bf9d-932e77d0b505/volumes" Jul 31 14:58:18.285679 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:18.285643 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:22.959725 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:22.959685 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:59.480202 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:59.480168 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:58:59.980784 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:58:59.980748 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:05.676544 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:05.676494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:05.778379 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:05.778343 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:05.964279 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:05.964197 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:06.206615 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:06.206582 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:07.404309 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:07.404267 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:17.566961 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:17.566894 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:18.863733 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:18.863666 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:19.950315 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:19.950281 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:26.280772 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:26.280722 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:26.338702 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:26.338667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:26.981469 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:26.981432 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:32.553798 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:32.553765 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:32.613940 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:32.613893 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:32.690725 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:32.690689 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:32.791291 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:32.791261 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:43.321864 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:43.321827 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 14:59:43.544413 ip-10-0-130-33 kubenswrapper[2558]: I0731 14:59:43.544376 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:04.708613 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:04.708573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:10.578304 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:10.578223 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:10.645471 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:10.645439 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:10.942877 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:10.942842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:11.243820 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:11.243721 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:11.487269 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:11.487230 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:14.909973 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.909935 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h"] Jul 31 15:00:14.910540 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.910520 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9056bf3a-6019-4689-bf9d-932e77d0b505" containerName="maas-api" Jul 31 15:00:14.910586 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.910544 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9056bf3a-6019-4689-bf9d-932e77d0b505" containerName="maas-api" Jul 31 15:00:14.910652 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.910641 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9056bf3a-6019-4689-bf9d-932e77d0b505" containerName="maas-api" Jul 31 15:00:14.913423 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.913398 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:14.916404 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.916382 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-55959d-serving-cert\"" Jul 31 15:00:14.923420 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.923393 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h"] Jul 31 15:00:14.990122 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.990091 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2333178-fbb0-4e99-9c0e-7a754d35802e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:14.990291 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.990140 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:14.990291 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:14.990221 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqc6f\" (UniqueName: \"kubernetes.io/projected/a2333178-fbb0-4e99-9c0e-7a754d35802e-kube-api-access-qqc6f\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.091265 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.091230 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.091453 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.091311 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqc6f\" (UniqueName: \"kubernetes.io/projected/a2333178-fbb0-4e99-9c0e-7a754d35802e-kube-api-access-qqc6f\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.091453 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:00:15.091394 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-serving-cert: secret "maas-api-e2e-inf-a-55959d-serving-cert" not found Jul 31 15:00:15.091453 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.091419 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2333178-fbb0-4e99-9c0e-7a754d35802e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.091607 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:00:15.091466 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls podName:a2333178-fbb0-4e99-9c0e-7a754d35802e nodeName:}" failed. No retries permitted until 2026-07-31 15:00:15.591445727 +0000 UTC m=+2881.099824793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls") pod "maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" (UID: "a2333178-fbb0-4e99-9c0e-7a754d35802e") : secret "maas-api-e2e-inf-a-55959d-serving-cert" not found Jul 31 15:00:15.092222 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.092202 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2333178-fbb0-4e99-9c0e-7a754d35802e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.111947 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.111924 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqc6f\" (UniqueName: \"kubernetes.io/projected/a2333178-fbb0-4e99-9c0e-7a754d35802e-kube-api-access-qqc6f\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.168056 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.167962 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:15.284420 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.284383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:15.385141 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.385110 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:15.597229 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.597130 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.599648 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.599618 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls\") pod \"maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.613093 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.613064 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:15.712229 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.712198 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:15.827308 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.827271 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:15.980978 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:15.980946 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h"] Jul 31 15:00:15.982160 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:00:15.982134 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2333178_fbb0_4e99_9c0e_7a754d35802e.slice/crio-467c3f8d362c6aed2a46f036a0cc7bc7e8ead122aafac8c90d16b02ae3b7bc91 WatchSource:0}: Error finding container 467c3f8d362c6aed2a46f036a0cc7bc7e8ead122aafac8c90d16b02ae3b7bc91: Status 404 returned error can't find the container with id 467c3f8d362c6aed2a46f036a0cc7bc7e8ead122aafac8c90d16b02ae3b7bc91 Jul 31 15:00:16.108614 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:16.108577 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" event={"ID":"a2333178-fbb0-4e99-9c0e-7a754d35802e","Type":"ContainerStarted","Data":"467c3f8d362c6aed2a46f036a0cc7bc7e8ead122aafac8c90d16b02ae3b7bc91"} Jul 31 15:00:17.115258 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:17.115214 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" event={"ID":"a2333178-fbb0-4e99-9c0e-7a754d35802e","Type":"ContainerStarted","Data":"fabf7edeec930d7d7108ce5bb114dc58074960ab34e2f66662e0ad2ba631a56a"} Jul 31 15:00:17.115664 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:17.115341 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:17.134584 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:17.134533 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" podStartSLOduration=2.70124544 podStartE2EDuration="3.134515096s" totalImagesPullingTime="433.269656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:17.132305937 +0000 UTC m=+2882.640685005" watchObservedRunningTime="2026-07-31 15:00:17.134515096 +0000 UTC m=+2882.642894161" Jul 31 15:00:23.126676 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:23.126647 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:00:38.858941 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:38.858884 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:39.184029 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:39.183995 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:39.497307 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:39.497206 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:39.967345 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:39.967315 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:41.942404 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:41.942367 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:42.043960 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:42.043922 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:43.009674 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.009640 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw"] Jul 31 15:00:43.013648 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.013626 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.016376 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.016349 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ef4b6d-serving-cert\"" Jul 31 15:00:43.021975 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.021895 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw"] Jul 31 15:00:43.038537 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.038510 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:43.147855 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.147819 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.148047 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.147880 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.148047 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.147953 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ql2k\" (UniqueName: \"kubernetes.io/projected/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-kube-api-access-2ql2k\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.249321 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.249287 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.249499 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.249361 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.249499 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.249394 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ql2k\" (UniqueName: \"kubernetes.io/projected/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-kube-api-access-2ql2k\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.249499 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:00:43.249458 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-serving-cert: secret "maas-api-e2e-inf-b-ef4b6d-serving-cert" not found Jul 31 15:00:43.249654 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:00:43.249525 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls podName:0daa3b6f-9843-4722-a405-7ce4d34a1b8b nodeName:}" failed. No retries permitted until 2026-07-31 15:00:43.749504231 +0000 UTC m=+2909.257883289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls") pod "maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" (UID: "0daa3b6f-9843-4722-a405-7ce4d34a1b8b") : secret "maas-api-e2e-inf-b-ef4b6d-serving-cert" not found Jul 31 15:00:43.250049 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.250030 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.258186 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.258159 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ql2k\" (UniqueName: \"kubernetes.io/projected/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-kube-api-access-2ql2k\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.440817 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.440783 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:43.541963 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.541929 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:43.604228 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.604194 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:00:43.609059 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.609035 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg"] Jul 31 15:00:43.612718 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.612698 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:43.615748 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.615727 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ef4b6d\"" Jul 31 15:00:43.615846 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.615727 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ef4b6d-dockercfg-zmr7t\"" Jul 31 15:00:43.621576 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.621554 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg"] Jul 31 15:00:43.656318 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.656290 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm"] Jul 31 15:00:43.659649 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.659635 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:43.669852 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.669830 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm"] Jul 31 15:00:43.754259 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.754159 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2zt\" (UniqueName: \"kubernetes.io/projected/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-kube-api-access-rh2zt\") pod \"payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg\" (UID: \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:43.754259 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.754207 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.754259 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.754232 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540e453-20ce-4ae1-b31f-986a17b8567c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm\" (UID: \"9540e453-20ce-4ae1-b31f-986a17b8567c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:43.754563 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.754263 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9q4h\" (UniqueName: \"kubernetes.io/projected/9540e453-20ce-4ae1-b31f-986a17b8567c-kube-api-access-r9q4h\") pod \"payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm\" (UID: \"9540e453-20ce-4ae1-b31f-986a17b8567c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:43.754563 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.754286 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg\" (UID: \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:43.756879 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.756841 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.855460 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.855416 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2zt\" (UniqueName: \"kubernetes.io/projected/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-kube-api-access-rh2zt\") pod \"payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg\" (UID: \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:43.855641 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.855474 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540e453-20ce-4ae1-b31f-986a17b8567c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm\" (UID: \"9540e453-20ce-4ae1-b31f-986a17b8567c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:43.855641 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.855507 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9q4h\" (UniqueName: \"kubernetes.io/projected/9540e453-20ce-4ae1-b31f-986a17b8567c-kube-api-access-r9q4h\") pod \"payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm\" (UID: \"9540e453-20ce-4ae1-b31f-986a17b8567c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:43.855641 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.855533 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg\" (UID: \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:43.856250 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.856229 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540e453-20ce-4ae1-b31f-986a17b8567c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm\" (UID: \"9540e453-20ce-4ae1-b31f-986a17b8567c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:43.856483 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.856461 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg\" (UID: \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:43.865013 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.864988 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9q4h\" (UniqueName: \"kubernetes.io/projected/9540e453-20ce-4ae1-b31f-986a17b8567c-kube-api-access-r9q4h\") pod \"payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm\" (UID: \"9540e453-20ce-4ae1-b31f-986a17b8567c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:43.865128 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.865108 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2zt\" (UniqueName: \"kubernetes.io/projected/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-kube-api-access-rh2zt\") pod \"payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg\" (UID: \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:43.925750 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.925700 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:43.929494 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.929461 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:43.970755 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:43.970718 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:44.111884 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:44.111794 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw"] Jul 31 15:00:44.112481 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:00:44.112451 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0daa3b6f_9843_4722_a405_7ce4d34a1b8b.slice/crio-3b874c9707149fcc2473c1634e6c6c34ee58396214882a15341b861e71f9844d WatchSource:0}: Error finding container 3b874c9707149fcc2473c1634e6c6c34ee58396214882a15341b861e71f9844d: Status 404 returned error can't find the container with id 3b874c9707149fcc2473c1634e6c6c34ee58396214882a15341b861e71f9844d Jul 31 15:00:44.140654 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:44.140625 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm"] Jul 31 15:00:44.142029 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:00:44.141975 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9540e453_20ce_4ae1_b31f_986a17b8567c.slice/crio-f6a1fb880fb1d7582379221ecc56ea4893dfb20739d0e46f07b74d5c16b9de04 WatchSource:0}: Error finding container f6a1fb880fb1d7582379221ecc56ea4893dfb20739d0e46f07b74d5c16b9de04: Status 404 returned error can't find the container with id f6a1fb880fb1d7582379221ecc56ea4893dfb20739d0e46f07b74d5c16b9de04 Jul 31 15:00:44.261771 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:44.261735 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" event={"ID":"9540e453-20ce-4ae1-b31f-986a17b8567c","Type":"ContainerStarted","Data":"9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7"} Jul 31 15:00:44.261954 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:44.261794 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" event={"ID":"9540e453-20ce-4ae1-b31f-986a17b8567c","Type":"ContainerStarted","Data":"f6a1fb880fb1d7582379221ecc56ea4893dfb20739d0e46f07b74d5c16b9de04"} Jul 31 15:00:44.261954 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:44.261823 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:44.263428 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:44.263403 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" event={"ID":"0daa3b6f-9843-4722-a405-7ce4d34a1b8b","Type":"ContainerStarted","Data":"3b874c9707149fcc2473c1634e6c6c34ee58396214882a15341b861e71f9844d"} Jul 31 15:00:44.279306 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:44.279216 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" podStartSLOduration=1.279198056 podStartE2EDuration="1.279198056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:44.278572376 +0000 UTC m=+2909.786951442" watchObservedRunningTime="2026-07-31 15:00:44.279198056 +0000 UTC m=+2909.787577125" Jul 31 15:00:44.295794 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:44.295769 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg"] Jul 31 15:00:44.297352 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:00:44.297327 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7625b5_7ee3_4d91_a74c_7d844b7910cc.slice/crio-8a65161e3de2ab18958743495b28fd19e5c5a6e925ad6ec95e7d160469507f20 WatchSource:0}: Error finding container 8a65161e3de2ab18958743495b28fd19e5c5a6e925ad6ec95e7d160469507f20: Status 404 returned error can't find the container with id 8a65161e3de2ab18958743495b28fd19e5c5a6e925ad6ec95e7d160469507f20 Jul 31 15:00:45.279516 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:45.279480 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" event={"ID":"0daa3b6f-9843-4722-a405-7ce4d34a1b8b","Type":"ContainerStarted","Data":"6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64"} Jul 31 15:00:45.279988 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:45.279673 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:45.281074 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:45.281053 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" event={"ID":"5d7625b5-7ee3-4d91-a74c-7d844b7910cc","Type":"ContainerStarted","Data":"cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676"} Jul 31 15:00:45.281215 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:45.281090 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" event={"ID":"5d7625b5-7ee3-4d91-a74c-7d844b7910cc","Type":"ContainerStarted","Data":"8a65161e3de2ab18958743495b28fd19e5c5a6e925ad6ec95e7d160469507f20"} Jul 31 15:00:45.281283 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:45.281269 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:00:45.296964 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:45.296895 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" podStartSLOduration=2.976107175 podStartE2EDuration="3.296878996s" totalImagesPullingTime="320.771821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:45.29586109 +0000 UTC m=+2910.804240156" watchObservedRunningTime="2026-07-31 15:00:45.296878996 +0000 UTC m=+2910.805258063" Jul 31 15:00:45.313153 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:45.313109 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" podStartSLOduration=2.313088941 podStartE2EDuration="2.313088941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:45.311279623 +0000 UTC m=+2910.819658691" watchObservedRunningTime="2026-07-31 15:00:45.313088941 +0000 UTC m=+2910.821468006" Jul 31 15:00:51.293555 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:51.293528 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:00:55.283407 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:55.283373 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:00:56.289401 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:00:56.289370 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:01:07.534717 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:01:07.534680 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:16.498017 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:01:16.497979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:01:42.195215 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:01:42.195114 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:26.940848 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:26.940813 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:28.098994 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:28.098956 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:38.653611 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.653577 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h"] Jul 31 15:02:38.654157 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.653813 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" podUID="a2333178-fbb0-4e99-9c0e-7a754d35802e" containerName="maas-api" containerID="cri-o://fabf7edeec930d7d7108ce5bb114dc58074960ab34e2f66662e0ad2ba631a56a" gracePeriod=30 Jul 31 15:02:38.676189 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.676161 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:38.774625 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.774601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:38.905172 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.905112 2558 generic.go:356] "Generic (PLEG): container finished" podID="a2333178-fbb0-4e99-9c0e-7a754d35802e" containerID="fabf7edeec930d7d7108ce5bb114dc58074960ab34e2f66662e0ad2ba631a56a" exitCode=0 Jul 31 15:02:38.905318 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.905185 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" event={"ID":"a2333178-fbb0-4e99-9c0e-7a754d35802e","Type":"ContainerDied","Data":"fabf7edeec930d7d7108ce5bb114dc58074960ab34e2f66662e0ad2ba631a56a"} Jul 31 15:02:38.905318 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.905253 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" event={"ID":"a2333178-fbb0-4e99-9c0e-7a754d35802e","Type":"ContainerDied","Data":"467c3f8d362c6aed2a46f036a0cc7bc7e8ead122aafac8c90d16b02ae3b7bc91"} Jul 31 15:02:38.905318 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.905299 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="467c3f8d362c6aed2a46f036a0cc7bc7e8ead122aafac8c90d16b02ae3b7bc91" Jul 31 15:02:38.908552 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.908532 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:38.915200 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.915174 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:02:38.988098 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.988072 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2333178-fbb0-4e99-9c0e-7a754d35802e-openshift-service-ca\") pod \"a2333178-fbb0-4e99-9c0e-7a754d35802e\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " Jul 31 15:02:38.988281 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.988134 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqc6f\" (UniqueName: \"kubernetes.io/projected/a2333178-fbb0-4e99-9c0e-7a754d35802e-kube-api-access-qqc6f\") pod \"a2333178-fbb0-4e99-9c0e-7a754d35802e\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " Jul 31 15:02:38.988281 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.988203 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls\") pod \"a2333178-fbb0-4e99-9c0e-7a754d35802e\" (UID: \"a2333178-fbb0-4e99-9c0e-7a754d35802e\") " Jul 31 15:02:38.988451 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.988423 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2333178-fbb0-4e99-9c0e-7a754d35802e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2333178-fbb0-4e99-9c0e-7a754d35802e" (UID: "a2333178-fbb0-4e99-9c0e-7a754d35802e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:02:38.990284 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.990261 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2333178-fbb0-4e99-9c0e-7a754d35802e" (UID: "a2333178-fbb0-4e99-9c0e-7a754d35802e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:02:38.990381 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:38.990320 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2333178-fbb0-4e99-9c0e-7a754d35802e-kube-api-access-qqc6f" (OuterVolumeSpecName: "kube-api-access-qqc6f") pod "a2333178-fbb0-4e99-9c0e-7a754d35802e" (UID: "a2333178-fbb0-4e99-9c0e-7a754d35802e"). InnerVolumeSpecName "kube-api-access-qqc6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:39.000230 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:39.000209 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:39.089999 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:39.089963 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2333178-fbb0-4e99-9c0e-7a754d35802e-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:39.089999 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:39.089993 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2333178-fbb0-4e99-9c0e-7a754d35802e-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:39.090204 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:39.090021 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqc6f\" (UniqueName: \"kubernetes.io/projected/a2333178-fbb0-4e99-9c0e-7a754d35802e-kube-api-access-qqc6f\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:39.388502 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:39.388462 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:39.910801 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:39.910769 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h" Jul 31 15:02:39.932402 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:39.932375 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h"] Jul 31 15:02:39.936140 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:39.936120 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55959d-5b994f87fd-bj57h"] Jul 31 15:02:40.983587 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:40.983546 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2333178-fbb0-4e99-9c0e-7a754d35802e" path="/var/lib/kubelet/pods/a2333178-fbb0-4e99-9c0e-7a754d35802e/volumes" Jul 31 15:02:45.180856 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:45.180820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:45.282339 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:45.282302 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:45.469393 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:45.469305 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:45.689098 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:45.689064 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:56.132277 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.132225 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw"] Jul 31 15:02:56.133088 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.133056 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" podUID="0daa3b6f-9843-4722-a405-7ce4d34a1b8b" containerName="maas-api" containerID="cri-o://6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64" gracePeriod=30 Jul 31 15:02:56.155223 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.155192 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:56.182217 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.182186 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm"] Jul 31 15:02:56.182693 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.182586 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" podUID="9540e453-20ce-4ae1-b31f-986a17b8567c" containerName="payload-processing" containerID="cri-o://9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7" gracePeriod=30 Jul 31 15:02:56.187833 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.187801 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg"] Jul 31 15:02:56.188096 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.188068 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" podUID="5d7625b5-7ee3-4d91-a74c-7d844b7910cc" containerName="payload-pre-processing" containerID="cri-o://cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676" gracePeriod=30 Jul 31 15:02:56.222830 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.222776 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:56.287793 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.287739 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" podUID="5d7625b5-7ee3-4d91-a74c-7d844b7910cc" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.62:9004: connect: connection refused" Jul 31 15:02:56.287946 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.287802 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" podUID="0daa3b6f-9843-4722-a405-7ce4d34a1b8b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.61:8443/health\": dial tcp 10.132.0.61:8443: connect: connection refused" Jul 31 15:02:56.289486 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.289431 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:56.342951 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.342918 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:56.459329 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.459306 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:02:56.462753 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.462727 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:02:56.486664 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.486640 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:02:56.558970 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.558942 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ql2k\" (UniqueName: \"kubernetes.io/projected/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-kube-api-access-2ql2k\") pod \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " Jul 31 15:02:56.559131 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559003 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls\") pod \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " Jul 31 15:02:56.559131 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559047 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-plugins-config-volume\") pod \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\" (UID: \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\") " Jul 31 15:02:56.559131 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559080 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-openshift-service-ca\") pod \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\" (UID: \"0daa3b6f-9843-4722-a405-7ce4d34a1b8b\") " Jul 31 15:02:56.559131 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559108 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540e453-20ce-4ae1-b31f-986a17b8567c-plugins-config-volume\") pod \"9540e453-20ce-4ae1-b31f-986a17b8567c\" (UID: \"9540e453-20ce-4ae1-b31f-986a17b8567c\") " Jul 31 15:02:56.559354 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559169 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9q4h\" (UniqueName: \"kubernetes.io/projected/9540e453-20ce-4ae1-b31f-986a17b8567c-kube-api-access-r9q4h\") pod \"9540e453-20ce-4ae1-b31f-986a17b8567c\" (UID: \"9540e453-20ce-4ae1-b31f-986a17b8567c\") " Jul 31 15:02:56.559354 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559216 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh2zt\" (UniqueName: \"kubernetes.io/projected/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-kube-api-access-rh2zt\") pod \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\" (UID: \"5d7625b5-7ee3-4d91-a74c-7d844b7910cc\") " Jul 31 15:02:56.559486 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559455 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0daa3b6f-9843-4722-a405-7ce4d34a1b8b" (UID: "0daa3b6f-9843-4722-a405-7ce4d34a1b8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:02:56.559542 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559500 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d7625b5-7ee3-4d91-a74c-7d844b7910cc" (UID: "5d7625b5-7ee3-4d91-a74c-7d844b7910cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:02:56.559589 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.559503 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9540e453-20ce-4ae1-b31f-986a17b8567c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9540e453-20ce-4ae1-b31f-986a17b8567c" (UID: "9540e453-20ce-4ae1-b31f-986a17b8567c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:02:56.561197 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.561174 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0daa3b6f-9843-4722-a405-7ce4d34a1b8b" (UID: "0daa3b6f-9843-4722-a405-7ce4d34a1b8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:02:56.561341 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.561223 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-kube-api-access-2ql2k" (OuterVolumeSpecName: "kube-api-access-2ql2k") pod "0daa3b6f-9843-4722-a405-7ce4d34a1b8b" (UID: "0daa3b6f-9843-4722-a405-7ce4d34a1b8b"). InnerVolumeSpecName "kube-api-access-2ql2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:56.561341 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.561257 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-kube-api-access-rh2zt" (OuterVolumeSpecName: "kube-api-access-rh2zt") pod "5d7625b5-7ee3-4d91-a74c-7d844b7910cc" (UID: "5d7625b5-7ee3-4d91-a74c-7d844b7910cc"). InnerVolumeSpecName "kube-api-access-rh2zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:56.561415 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.561352 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9540e453-20ce-4ae1-b31f-986a17b8567c-kube-api-access-r9q4h" (OuterVolumeSpecName: "kube-api-access-r9q4h") pod "9540e453-20ce-4ae1-b31f-986a17b8567c" (UID: "9540e453-20ce-4ae1-b31f-986a17b8567c"). InnerVolumeSpecName "kube-api-access-r9q4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:02:56.660762 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.660685 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9q4h\" (UniqueName: \"kubernetes.io/projected/9540e453-20ce-4ae1-b31f-986a17b8567c-kube-api-access-r9q4h\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:56.660762 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.660714 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh2zt\" (UniqueName: \"kubernetes.io/projected/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-kube-api-access-rh2zt\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:56.660762 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.660724 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ql2k\" (UniqueName: \"kubernetes.io/projected/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-kube-api-access-2ql2k\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:56.660762 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.660733 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:56.660762 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.660741 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7625b5-7ee3-4d91-a74c-7d844b7910cc-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:56.660762 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.660750 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0daa3b6f-9843-4722-a405-7ce4d34a1b8b-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:56.660762 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.660758 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540e453-20ce-4ae1-b31f-986a17b8567c-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:02:56.742374 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:56.742331 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:02:57.003314 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.003287 2558 generic.go:356] "Generic (PLEG): container finished" podID="0daa3b6f-9843-4722-a405-7ce4d34a1b8b" containerID="6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64" exitCode=0 Jul 31 15:02:57.003425 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.003328 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" Jul 31 15:02:57.003425 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.003360 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" event={"ID":"0daa3b6f-9843-4722-a405-7ce4d34a1b8b","Type":"ContainerDied","Data":"6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64"} Jul 31 15:02:57.003425 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.003417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw" event={"ID":"0daa3b6f-9843-4722-a405-7ce4d34a1b8b","Type":"ContainerDied","Data":"3b874c9707149fcc2473c1634e6c6c34ee58396214882a15341b861e71f9844d"} Jul 31 15:02:57.003548 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.003447 2558 scope.go:117] "RemoveContainer" containerID="6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64" Jul 31 15:02:57.004778 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.004756 2558 generic.go:356] "Generic (PLEG): container finished" podID="5d7625b5-7ee3-4d91-a74c-7d844b7910cc" containerID="cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676" exitCode=0 Jul 31 15:02:57.004872 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.004796 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" Jul 31 15:02:57.004872 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.004851 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" event={"ID":"5d7625b5-7ee3-4d91-a74c-7d844b7910cc","Type":"ContainerDied","Data":"cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676"} Jul 31 15:02:57.005012 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.004897 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg" event={"ID":"5d7625b5-7ee3-4d91-a74c-7d844b7910cc","Type":"ContainerDied","Data":"8a65161e3de2ab18958743495b28fd19e5c5a6e925ad6ec95e7d160469507f20"} Jul 31 15:02:57.006592 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.006575 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" Jul 31 15:02:57.006715 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.006591 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" event={"ID":"9540e453-20ce-4ae1-b31f-986a17b8567c","Type":"ContainerDied","Data":"9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7"} Jul 31 15:02:57.006715 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.006530 2558 generic.go:356] "Generic (PLEG): container finished" podID="9540e453-20ce-4ae1-b31f-986a17b8567c" containerID="9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7" exitCode=0 Jul 31 15:02:57.006827 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.006792 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm" event={"ID":"9540e453-20ce-4ae1-b31f-986a17b8567c","Type":"ContainerDied","Data":"f6a1fb880fb1d7582379221ecc56ea4893dfb20739d0e46f07b74d5c16b9de04"} Jul 31 15:02:57.014231 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.014211 2558 scope.go:117] "RemoveContainer" containerID="6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64" Jul 31 15:02:57.014516 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:02:57.014499 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64\": container with ID starting with 6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64 not found: ID does not exist" containerID="6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64" Jul 31 15:02:57.014593 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.014530 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64"} err="failed to get container status \"6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64\": rpc error: code = NotFound desc = could not find container \"6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64\": container with ID starting with 6aca2006c46f8de2597f2b7c042a2bf02425b7ed16af488823a1726a66165f64 not found: ID does not exist" Jul 31 15:02:57.014593 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.014546 2558 scope.go:117] "RemoveContainer" containerID="cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676" Jul 31 15:02:57.022568 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.022554 2558 scope.go:117] "RemoveContainer" containerID="cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676" Jul 31 15:02:57.022801 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:02:57.022784 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676\": container with ID starting with cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676 not found: ID does not exist" containerID="cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676" Jul 31 15:02:57.022847 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.022805 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676"} err="failed to get container status \"cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676\": rpc error: code = NotFound desc = could not find container \"cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676\": container with ID starting with cab68d37ee6039e4083a74a672d248738e7cab80f0583f6961ea43635b21c676 not found: ID does not exist" Jul 31 15:02:57.022847 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.022823 2558 scope.go:117] "RemoveContainer" containerID="9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7" Jul 31 15:02:57.028374 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.028356 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg"] Jul 31 15:02:57.031603 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.031588 2558 scope.go:117] "RemoveContainer" containerID="9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7" Jul 31 15:02:57.031850 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:02:57.031821 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7\": container with ID starting with 9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7 not found: ID does not exist" containerID="9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7" Jul 31 15:02:57.031926 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.031858 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7"} err="failed to get container status \"9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7\": rpc error: code = NotFound desc = could not find container \"9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7\": container with ID starting with 9d68afa3912aaf5e8d7cdcf48761ffdd1d83cd505cc9637d580a6f7c39d744f7 not found: ID does not exist" Jul 31 15:02:57.032446 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.032430 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef4b6d-6d945bc6f6-fbbmg"] Jul 31 15:02:57.043271 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.043249 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw"] Jul 31 15:02:57.049489 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.049465 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef4b6d-84f95779c5-smhhw"] Jul 31 15:02:57.063668 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.062399 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm"] Jul 31 15:02:57.067698 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:57.067672 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef4b6d-7f4bff5ffd-j82nm"] Jul 31 15:02:58.983451 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:58.983420 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0daa3b6f-9843-4722-a405-7ce4d34a1b8b" path="/var/lib/kubelet/pods/0daa3b6f-9843-4722-a405-7ce4d34a1b8b/volumes" Jul 31 15:02:58.983832 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:58.983773 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7625b5-7ee3-4d91-a74c-7d844b7910cc" path="/var/lib/kubelet/pods/5d7625b5-7ee3-4d91-a74c-7d844b7910cc/volumes" Jul 31 15:02:58.984138 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:02:58.984124 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9540e453-20ce-4ae1-b31f-986a17b8567c" path="/var/lib/kubelet/pods/9540e453-20ce-4ae1-b31f-986a17b8567c/volumes" Jul 31 15:03:02.384209 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:02.384162 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:02.484107 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:02.484072 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:02.668412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:02.668321 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:02.899249 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:02.899211 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:03.266151 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:03.266118 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:03.571374 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:03.571300 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:03.688952 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:03.688898 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:06.593830 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:06.593785 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:06.693523 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:06.693473 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:07.730169 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730093 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x"] Jul 31 15:03:07.730670 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730654 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0daa3b6f-9843-4722-a405-7ce4d34a1b8b" containerName="maas-api" Jul 31 15:03:07.730713 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730674 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0daa3b6f-9843-4722-a405-7ce4d34a1b8b" containerName="maas-api" Jul 31 15:03:07.730713 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730690 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2333178-fbb0-4e99-9c0e-7a754d35802e" containerName="maas-api" Jul 31 15:03:07.730713 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730700 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2333178-fbb0-4e99-9c0e-7a754d35802e" containerName="maas-api" Jul 31 15:03:07.730811 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730712 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7625b5-7ee3-4d91-a74c-7d844b7910cc" containerName="payload-pre-processing" Jul 31 15:03:07.730811 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730721 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7625b5-7ee3-4d91-a74c-7d844b7910cc" containerName="payload-pre-processing" Jul 31 15:03:07.730811 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730734 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9540e453-20ce-4ae1-b31f-986a17b8567c" containerName="payload-processing" Jul 31 15:03:07.730811 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730743 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9540e453-20ce-4ae1-b31f-986a17b8567c" containerName="payload-processing" Jul 31 15:03:07.730961 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730858 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7625b5-7ee3-4d91-a74c-7d844b7910cc" containerName="payload-pre-processing" Jul 31 15:03:07.730961 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730871 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0daa3b6f-9843-4722-a405-7ce4d34a1b8b" containerName="maas-api" Jul 31 15:03:07.730961 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730882 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2333178-fbb0-4e99-9c0e-7a754d35802e" containerName="maas-api" Jul 31 15:03:07.730961 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.730892 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9540e453-20ce-4ae1-b31f-986a17b8567c" containerName="payload-processing" Jul 31 15:03:07.735829 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.735808 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.738360 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.738332 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-700618-serving-cert\"" Jul 31 15:03:07.743651 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.743628 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x"] Jul 31 15:03:07.760677 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.760648 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:07.865687 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.865652 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97dd\" (UniqueName: \"kubernetes.io/projected/76096cc7-2bb2-4848-b666-6c6c639ddf07-kube-api-access-l97dd\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.865687 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.865697 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76096cc7-2bb2-4848-b666-6c6c639ddf07-openshift-service-ca\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.865957 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.865857 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76096cc7-2bb2-4848-b666-6c6c639ddf07-maas-api-tls\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.962645 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.962611 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:07.966838 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.966813 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76096cc7-2bb2-4848-b666-6c6c639ddf07-maas-api-tls\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.966934 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.966852 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97dd\" (UniqueName: \"kubernetes.io/projected/76096cc7-2bb2-4848-b666-6c6c639ddf07-kube-api-access-l97dd\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.966934 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.966878 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76096cc7-2bb2-4848-b666-6c6c639ddf07-openshift-service-ca\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.967561 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.967540 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76096cc7-2bb2-4848-b666-6c6c639ddf07-openshift-service-ca\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.969223 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.969200 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76096cc7-2bb2-4848-b666-6c6c639ddf07-maas-api-tls\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:07.975378 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:07.975356 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97dd\" (UniqueName: \"kubernetes.io/projected/76096cc7-2bb2-4848-b666-6c6c639ddf07-kube-api-access-l97dd\") pod \"maas-api-e2e-api-a-700618-86845877fd-m4m5x\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:08.049464 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:08.049370 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:08.193629 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:08.193599 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:08.199825 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:08.199795 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x"] Jul 31 15:03:08.201374 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:03:08.201327 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76096cc7_2bb2_4848_b666_6c6c639ddf07.slice/crio-49ee3e9d73bd2a9ab9e219b3fb281c4c71e4fc281a164ecb4a380ae964cc8e66 WatchSource:0}: Error finding container 49ee3e9d73bd2a9ab9e219b3fb281c4c71e4fc281a164ecb4a380ae964cc8e66: Status 404 returned error can't find the container with id 49ee3e9d73bd2a9ab9e219b3fb281c4c71e4fc281a164ecb4a380ae964cc8e66 Jul 31 15:03:08.202860 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:08.202840 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 15:03:08.294285 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:08.294246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:09.075317 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:09.075229 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" event={"ID":"76096cc7-2bb2-4848-b666-6c6c639ddf07","Type":"ContainerStarted","Data":"dbd3834e7651d7d70b8fb6f4f53c0a9356bb504eb12a3d43dffecedf257461c9"} Jul 31 15:03:09.075317 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:09.075278 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" event={"ID":"76096cc7-2bb2-4848-b666-6c6c639ddf07","Type":"ContainerStarted","Data":"49ee3e9d73bd2a9ab9e219b3fb281c4c71e4fc281a164ecb4a380ae964cc8e66"} Jul 31 15:03:09.075722 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:09.075372 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:09.094248 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:09.094196 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" podStartSLOduration=1.628286545 podStartE2EDuration="2.094182711s" totalImagesPullingTime="465.896166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:09.09277634 +0000 UTC m=+3054.601155405" watchObservedRunningTime="2026-07-31 15:03:09.094182711 +0000 UTC m=+3054.602561776" Jul 31 15:03:15.087016 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:15.086985 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:03:31.621428 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:31.621390 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:32.120891 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:32.120863 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:32.415899 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:32.415807 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:32.593695 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:32.593662 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:35.782559 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:35.782524 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:35.882592 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:35.882543 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:37.009189 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.009155 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:37.206497 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.206462 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:37.543384 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.543351 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:37.641318 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.641264 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:03:37.678323 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.678287 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh"] Jul 31 15:03:37.682443 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.682422 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:37.685479 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.685460 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2eeeef\"" Jul 31 15:03:37.685585 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.685518 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2eeeef-dockercfg-6bwmb\"" Jul 31 15:03:37.692528 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.692498 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh"] Jul 31 15:03:37.722091 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.722063 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f979218-827a-47c0-ac2a-4513e75f3146-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh\" (UID: \"6f979218-827a-47c0-ac2a-4513e75f3146\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:37.722202 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.722097 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqbq6\" (UniqueName: \"kubernetes.io/projected/6f979218-827a-47c0-ac2a-4513e75f3146-kube-api-access-gqbq6\") pod \"payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh\" (UID: \"6f979218-827a-47c0-ac2a-4513e75f3146\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:37.823436 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.823356 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f979218-827a-47c0-ac2a-4513e75f3146-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh\" (UID: \"6f979218-827a-47c0-ac2a-4513e75f3146\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:37.823436 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.823390 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqbq6\" (UniqueName: \"kubernetes.io/projected/6f979218-827a-47c0-ac2a-4513e75f3146-kube-api-access-gqbq6\") pod \"payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh\" (UID: \"6f979218-827a-47c0-ac2a-4513e75f3146\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:37.824073 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.824055 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f979218-827a-47c0-ac2a-4513e75f3146-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh\" (UID: \"6f979218-827a-47c0-ac2a-4513e75f3146\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:37.831625 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.831592 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqbq6\" (UniqueName: \"kubernetes.io/projected/6f979218-827a-47c0-ac2a-4513e75f3146-kube-api-access-gqbq6\") pod \"payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh\" (UID: \"6f979218-827a-47c0-ac2a-4513e75f3146\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:37.994406 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:37.994371 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:38.136288 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:38.136260 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh"] Jul 31 15:03:38.138576 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:03:38.138549 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f979218_827a_47c0_ac2a_4513e75f3146.slice/crio-7282816ca0f07c5f4cbfc2fa4440f1deeaaff3eca073e9dd5d29d0510b6dec92 WatchSource:0}: Error finding container 7282816ca0f07c5f4cbfc2fa4440f1deeaaff3eca073e9dd5d29d0510b6dec92: Status 404 returned error can't find the container with id 7282816ca0f07c5f4cbfc2fa4440f1deeaaff3eca073e9dd5d29d0510b6dec92 Jul 31 15:03:38.231461 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:38.231427 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" event={"ID":"6f979218-827a-47c0-ac2a-4513e75f3146","Type":"ContainerStarted","Data":"16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0"} Jul 31 15:03:38.231651 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:38.231486 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" event={"ID":"6f979218-827a-47c0-ac2a-4513e75f3146","Type":"ContainerStarted","Data":"7282816ca0f07c5f4cbfc2fa4440f1deeaaff3eca073e9dd5d29d0510b6dec92"} Jul 31 15:03:38.231651 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:38.231534 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:03:38.250680 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:38.250625 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" podStartSLOduration=1.250609799 podStartE2EDuration="1.250609799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:38.247926206 +0000 UTC m=+3083.756305275" watchObservedRunningTime="2026-07-31 15:03:38.250609799 +0000 UTC m=+3083.758988865" Jul 31 15:03:49.240026 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:03:49.239990 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:04:00.824057 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:00.824021 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:11.767651 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:11.767611 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:19.278434 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.278395 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x"] Jul 31 15:04:19.280760 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.278639 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" podUID="76096cc7-2bb2-4848-b666-6c6c639ddf07" containerName="maas-api" containerID="cri-o://dbd3834e7651d7d70b8fb6f4f53c0a9356bb504eb12a3d43dffecedf257461c9" gracePeriod=30 Jul 31 15:04:19.302624 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.302591 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:19.461619 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.461588 2558 generic.go:356] "Generic (PLEG): container finished" podID="76096cc7-2bb2-4848-b666-6c6c639ddf07" containerID="dbd3834e7651d7d70b8fb6f4f53c0a9356bb504eb12a3d43dffecedf257461c9" exitCode=0 Jul 31 15:04:19.461770 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.461649 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" event={"ID":"76096cc7-2bb2-4848-b666-6c6c639ddf07","Type":"ContainerDied","Data":"dbd3834e7651d7d70b8fb6f4f53c0a9356bb504eb12a3d43dffecedf257461c9"} Jul 31 15:04:19.540615 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.540545 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:04:19.610616 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.610579 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76096cc7-2bb2-4848-b666-6c6c639ddf07-openshift-service-ca\") pod \"76096cc7-2bb2-4848-b666-6c6c639ddf07\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " Jul 31 15:04:19.610616 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.610623 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l97dd\" (UniqueName: \"kubernetes.io/projected/76096cc7-2bb2-4848-b666-6c6c639ddf07-kube-api-access-l97dd\") pod \"76096cc7-2bb2-4848-b666-6c6c639ddf07\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " Jul 31 15:04:19.610847 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.610740 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76096cc7-2bb2-4848-b666-6c6c639ddf07-maas-api-tls\") pod \"76096cc7-2bb2-4848-b666-6c6c639ddf07\" (UID: \"76096cc7-2bb2-4848-b666-6c6c639ddf07\") " Jul 31 15:04:19.611098 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.611064 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76096cc7-2bb2-4848-b666-6c6c639ddf07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76096cc7-2bb2-4848-b666-6c6c639ddf07" (UID: "76096cc7-2bb2-4848-b666-6c6c639ddf07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:04:19.612853 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.612820 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76096cc7-2bb2-4848-b666-6c6c639ddf07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76096cc7-2bb2-4848-b666-6c6c639ddf07" (UID: "76096cc7-2bb2-4848-b666-6c6c639ddf07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:04:19.612997 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.612932 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76096cc7-2bb2-4848-b666-6c6c639ddf07-kube-api-access-l97dd" (OuterVolumeSpecName: "kube-api-access-l97dd") pod "76096cc7-2bb2-4848-b666-6c6c639ddf07" (UID: "76096cc7-2bb2-4848-b666-6c6c639ddf07"). InnerVolumeSpecName "kube-api-access-l97dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:19.712335 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.712280 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76096cc7-2bb2-4848-b666-6c6c639ddf07-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:04:19.712335 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.712330 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76096cc7-2bb2-4848-b666-6c6c639ddf07-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:04:19.712335 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:19.712340 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97dd\" (UniqueName: \"kubernetes.io/projected/76096cc7-2bb2-4848-b666-6c6c639ddf07-kube-api-access-l97dd\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:04:20.468517 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:20.468477 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" event={"ID":"76096cc7-2bb2-4848-b666-6c6c639ddf07","Type":"ContainerDied","Data":"49ee3e9d73bd2a9ab9e219b3fb281c4c71e4fc281a164ecb4a380ae964cc8e66"} Jul 31 15:04:20.468517 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:20.468499 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x" Jul 31 15:04:20.469004 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:20.468542 2558 scope.go:117] "RemoveContainer" containerID="dbd3834e7651d7d70b8fb6f4f53c0a9356bb504eb12a3d43dffecedf257461c9" Jul 31 15:04:20.490804 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:20.490747 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x"] Jul 31 15:04:20.493992 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:20.493964 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-700618-86845877fd-m4m5x"] Jul 31 15:04:20.983956 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:20.983900 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76096cc7-2bb2-4848-b666-6c6c639ddf07" path="/var/lib/kubelet/pods/76096cc7-2bb2-4848-b666-6c6c639ddf07/volumes" Jul 31 15:04:23.028428 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:23.028386 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:25.624206 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:25.624170 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:25.725268 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:25.725229 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:25.894001 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:25.893870 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:26.142211 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:26.142176 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:41.681021 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:41.680930 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:41.701047 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:41.701011 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh"] Jul 31 15:04:41.701414 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:41.701346 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" podUID="6f979218-827a-47c0-ac2a-4513e75f3146" containerName="payload-processing" containerID="cri-o://16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0" gracePeriod=30 Jul 31 15:04:41.779526 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:41.779491 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:41.835316 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:41.835276 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:41.961013 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:41.960990 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:04:42.120334 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.120308 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqbq6\" (UniqueName: \"kubernetes.io/projected/6f979218-827a-47c0-ac2a-4513e75f3146-kube-api-access-gqbq6\") pod \"6f979218-827a-47c0-ac2a-4513e75f3146\" (UID: \"6f979218-827a-47c0-ac2a-4513e75f3146\") " Jul 31 15:04:42.120506 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.120369 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f979218-827a-47c0-ac2a-4513e75f3146-plugins-config-volume\") pod \"6f979218-827a-47c0-ac2a-4513e75f3146\" (UID: \"6f979218-827a-47c0-ac2a-4513e75f3146\") " Jul 31 15:04:42.120685 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.120658 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f979218-827a-47c0-ac2a-4513e75f3146-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f979218-827a-47c0-ac2a-4513e75f3146" (UID: "6f979218-827a-47c0-ac2a-4513e75f3146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:04:42.122234 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.122212 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f979218-827a-47c0-ac2a-4513e75f3146-kube-api-access-gqbq6" (OuterVolumeSpecName: "kube-api-access-gqbq6") pod "6f979218-827a-47c0-ac2a-4513e75f3146" (UID: "6f979218-827a-47c0-ac2a-4513e75f3146"). InnerVolumeSpecName "kube-api-access-gqbq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:04:42.211657 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.211576 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:42.221872 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.221847 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqbq6\" (UniqueName: \"kubernetes.io/projected/6f979218-827a-47c0-ac2a-4513e75f3146-kube-api-access-gqbq6\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:04:42.221872 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.221872 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f979218-827a-47c0-ac2a-4513e75f3146-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:04:42.592445 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.592355 2558 generic.go:356] "Generic (PLEG): container finished" podID="6f979218-827a-47c0-ac2a-4513e75f3146" containerID="16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0" exitCode=0 Jul 31 15:04:42.592445 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.592398 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" Jul 31 15:04:42.592445 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.592426 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" event={"ID":"6f979218-827a-47c0-ac2a-4513e75f3146","Type":"ContainerDied","Data":"16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0"} Jul 31 15:04:42.592735 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.592478 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh" event={"ID":"6f979218-827a-47c0-ac2a-4513e75f3146","Type":"ContainerDied","Data":"7282816ca0f07c5f4cbfc2fa4440f1deeaaff3eca073e9dd5d29d0510b6dec92"} Jul 31 15:04:42.592735 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.592509 2558 scope.go:117] "RemoveContainer" containerID="16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0" Jul 31 15:04:42.602368 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.602350 2558 scope.go:117] "RemoveContainer" containerID="16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0" Jul 31 15:04:42.602624 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:04:42.602603 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0\": container with ID starting with 16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0 not found: ID does not exist" containerID="16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0" Jul 31 15:04:42.602733 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.602633 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0"} err="failed to get container status \"16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0\": rpc error: code = NotFound desc = could not find container \"16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0\": container with ID starting with 16bfed71df0812fa0227345b51730a67c527dd4d9501b15c21d9694ace1fdde0 not found: ID does not exist" Jul 31 15:04:42.614782 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.614759 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh"] Jul 31 15:04:42.622919 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.619594 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eeeef-6fd897644b-6mzmh"] Jul 31 15:04:42.983843 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:42.983809 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f979218-827a-47c0-ac2a-4513e75f3146" path="/var/lib/kubelet/pods/6f979218-827a-47c0-ac2a-4513e75f3146/volumes" Jul 31 15:04:47.883319 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:47.883282 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:47.937530 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:47.937501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:48.166923 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:48.166826 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:48.388351 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:48.388321 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:48.731343 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:48.731298 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:49.026960 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:49.026855 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:49.168064 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:49.168029 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:52.148252 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:52.148182 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:52.245295 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:52.245263 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:53.229868 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.229837 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx"] Jul 31 15:04:53.230262 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.230247 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f979218-827a-47c0-ac2a-4513e75f3146" containerName="payload-processing" Jul 31 15:04:53.230319 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.230264 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f979218-827a-47c0-ac2a-4513e75f3146" containerName="payload-processing" Jul 31 15:04:53.230319 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.230279 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76096cc7-2bb2-4848-b666-6c6c639ddf07" containerName="maas-api" Jul 31 15:04:53.230319 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.230284 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="76096cc7-2bb2-4848-b666-6c6c639ddf07" containerName="maas-api" Jul 31 15:04:53.230421 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.230347 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f979218-827a-47c0-ac2a-4513e75f3146" containerName="payload-processing" Jul 31 15:04:53.230421 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.230355 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="76096cc7-2bb2-4848-b666-6c6c639ddf07" containerName="maas-api" Jul 31 15:04:53.234793 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.234764 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.238403 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.238378 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-9966b2-serving-cert\"" Jul 31 15:04:53.244244 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.244219 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx"] Jul 31 15:04:53.261637 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.261613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:53.320385 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.320359 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae131da-a85a-4d47-8e5f-35506750dcbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.320534 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.320389 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4frc\" (UniqueName: \"kubernetes.io/projected/9ae131da-a85a-4d47-8e5f-35506750dcbd-kube-api-access-s4frc\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.320534 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.320415 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.421474 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.421437 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae131da-a85a-4d47-8e5f-35506750dcbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.421474 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.421476 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4frc\" (UniqueName: \"kubernetes.io/projected/9ae131da-a85a-4d47-8e5f-35506750dcbd-kube-api-access-s4frc\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.421712 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.421511 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.421712 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:04:53.421682 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-serving-cert: secret "maas-api-e2e-shared-a-9966b2-serving-cert" not found Jul 31 15:04:53.421820 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:04:53.421753 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls podName:9ae131da-a85a-4d47-8e5f-35506750dcbd nodeName:}" failed. No retries permitted until 2026-07-31 15:04:53.921731459 +0000 UTC m=+3159.430110518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls") pod "maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" (UID: "9ae131da-a85a-4d47-8e5f-35506750dcbd") : secret "maas-api-e2e-shared-a-9966b2-serving-cert" not found Jul 31 15:04:53.422088 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.422064 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae131da-a85a-4d47-8e5f-35506750dcbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.430053 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.430022 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4frc\" (UniqueName: \"kubernetes.io/projected/9ae131da-a85a-4d47-8e5f-35506750dcbd-kube-api-access-s4frc\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.728362 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.728334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:53.821166 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.821130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:04:53.926212 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.926175 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:53.928895 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:53.928863 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:54.146575 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:54.146545 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:54.286717 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:54.286692 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx"] Jul 31 15:04:54.288642 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:04:54.288614 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae131da_a85a_4d47_8e5f_35506750dcbd.slice/crio-4ef37e2dc9dff0f0e356bc0e6af548ef94475e4ff981ec083514ac744cab14ed WatchSource:0}: Error finding container 4ef37e2dc9dff0f0e356bc0e6af548ef94475e4ff981ec083514ac744cab14ed: Status 404 returned error can't find the container with id 4ef37e2dc9dff0f0e356bc0e6af548ef94475e4ff981ec083514ac744cab14ed Jul 31 15:04:54.659177 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:54.659141 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" event={"ID":"9ae131da-a85a-4d47-8e5f-35506750dcbd","Type":"ContainerStarted","Data":"4ef37e2dc9dff0f0e356bc0e6af548ef94475e4ff981ec083514ac744cab14ed"} Jul 31 15:04:57.676324 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:57.676286 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" event={"ID":"9ae131da-a85a-4d47-8e5f-35506750dcbd","Type":"ContainerStarted","Data":"dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c"} Jul 31 15:04:57.676717 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:57.676407 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:04:57.694736 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:04:57.694678 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" podStartSLOduration=1.6522240689999999 podStartE2EDuration="4.694663918s" totalImagesPullingTime="3.042439849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:57.692405293 +0000 UTC m=+3163.200784351" watchObservedRunningTime="2026-07-31 15:04:57.694663918 +0000 UTC m=+3163.203042983" Jul 31 15:05:03.687438 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:03.687404 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:05:07.662607 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:07.662573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:07.763540 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:07.763508 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:17.196814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:17.196778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:17.565736 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:17.565652 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:17.892239 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:17.892194 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:18.314295 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:18.314210 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:18.373808 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:18.373776 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:18.474032 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:18.473996 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:21.297116 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:21.297075 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:22.968570 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:22.968531 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh"] Jul 31 15:05:22.972884 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:22.972866 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:22.975827 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:22.975804 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-79a3ee-dockercfg-j89k5\"" Jul 31 15:05:22.975964 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:22.975809 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-79a3ee\"" Jul 31 15:05:22.985278 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:22.985257 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh"] Jul 31 15:05:23.002654 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.002628 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj"] Jul 31 15:05:23.006005 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.005985 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.018133 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.018110 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj"] Jul 31 15:05:23.101811 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.101781 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52pv\" (UniqueName: \"kubernetes.io/projected/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-kube-api-access-p52pv\") pod \"payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj\" (UID: \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.101993 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.101822 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rsh\" (UniqueName: \"kubernetes.io/projected/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-kube-api-access-84rsh\") pod \"payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh\" (UID: \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:23.101993 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.101863 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh\" (UID: \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:23.101993 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.101895 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj\" (UID: \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.203325 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.203292 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh\" (UID: \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:23.203501 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.203343 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj\" (UID: \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.203501 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.203402 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52pv\" (UniqueName: \"kubernetes.io/projected/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-kube-api-access-p52pv\") pod \"payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj\" (UID: \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.203501 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.203448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rsh\" (UniqueName: \"kubernetes.io/projected/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-kube-api-access-84rsh\") pod \"payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh\" (UID: \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:23.204091 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.204072 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh\" (UID: \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:23.204142 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.204101 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj\" (UID: \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.211531 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.211503 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rsh\" (UniqueName: \"kubernetes.io/projected/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-kube-api-access-84rsh\") pod \"payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh\" (UID: \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:23.211802 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.211780 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52pv\" (UniqueName: \"kubernetes.io/projected/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-kube-api-access-p52pv\") pod \"payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj\" (UID: \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.286763 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.286686 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:23.316360 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.316320 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.444206 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.444178 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh"] Jul 31 15:05:23.446330 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:05:23.446305 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fb15d3_82a5_47e1_833d_a8a1e97e45ee.slice/crio-0d83cb1dd2073154b9bf017fb2a41f14197ae5b016271a287572bdfb83ba45ca WatchSource:0}: Error finding container 0d83cb1dd2073154b9bf017fb2a41f14197ae5b016271a287572bdfb83ba45ca: Status 404 returned error can't find the container with id 0d83cb1dd2073154b9bf017fb2a41f14197ae5b016271a287572bdfb83ba45ca Jul 31 15:05:23.468754 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.468728 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj"] Jul 31 15:05:23.470398 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:05:23.470371 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ad3dc5_27d6_46b8_8d8e_10152795f1b7.slice/crio-951f93c8a035a1d0cb1c4216a98f38792e6796c6337ceca9d3291d451130119b WatchSource:0}: Error finding container 951f93c8a035a1d0cb1c4216a98f38792e6796c6337ceca9d3291d451130119b: Status 404 returned error can't find the container with id 951f93c8a035a1d0cb1c4216a98f38792e6796c6337ceca9d3291d451130119b Jul 31 15:05:23.830808 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.830770 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" event={"ID":"44ad3dc5-27d6-46b8-8d8e-10152795f1b7","Type":"ContainerStarted","Data":"71158d4ce2a7915b29e539b82a62101884e46337ff41efb8d2a5cc5f79eec8b8"} Jul 31 15:05:23.831019 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.830841 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:23.831019 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.830868 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" event={"ID":"44ad3dc5-27d6-46b8-8d8e-10152795f1b7","Type":"ContainerStarted","Data":"951f93c8a035a1d0cb1c4216a98f38792e6796c6337ceca9d3291d451130119b"} Jul 31 15:05:23.832100 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.832074 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" event={"ID":"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee","Type":"ContainerStarted","Data":"bf897e0a4095f12633ece79675b0b02ed1a30946f05d68c12c7efcb4e6541106"} Jul 31 15:05:23.832206 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.832119 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" event={"ID":"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee","Type":"ContainerStarted","Data":"0d83cb1dd2073154b9bf017fb2a41f14197ae5b016271a287572bdfb83ba45ca"} Jul 31 15:05:23.832206 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.832202 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:23.847827 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.847773 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" podStartSLOduration=1.847760351 podStartE2EDuration="1.847760351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:05:23.84679207 +0000 UTC m=+3189.355171136" watchObservedRunningTime="2026-07-31 15:05:23.847760351 +0000 UTC m=+3189.356139417" Jul 31 15:05:23.866041 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:23.865994 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" podStartSLOduration=1.865979753 podStartE2EDuration="1.865979753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:05:23.863057037 +0000 UTC m=+3189.371436102" watchObservedRunningTime="2026-07-31 15:05:23.865979753 +0000 UTC m=+3189.374358820" Jul 31 15:05:25.046292 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:25.046256 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:25.147482 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:25.147447 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:34.839299 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:34.839255 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" podUID="e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.67:9004: connect: connection refused" Jul 31 15:05:34.839768 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:34.839614 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:05:44.838837 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:44.838800 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:05:46.347746 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:46.347705 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:05:49.295892 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:05:49.295858 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:11.088752 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:06:11.088667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:06:34.407666 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:06:34.407632 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:15.368119 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:07:15.368091 2558 scope.go:117] "RemoveContainer" containerID="fabf7edeec930d7d7108ce5bb114dc58074960ab34e2f66662e0ad2ba631a56a" Jul 31 15:07:48.478258 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:07:48.478219 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:07:48.551190 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:07:48.551154 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:38.989817 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:09:38.989786 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:45.196019 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:09:45.195979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:09:46.204526 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:09:46.204493 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:03.094710 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:10:03.094671 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:10:24.873700 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:10:24.873665 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:26.568178 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:26.568139 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:37.359889 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:37.359852 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:38.274113 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:38.274074 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:52.392514 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.392481 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv"] Jul 31 15:11:52.396156 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.396137 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.400267 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.400247 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"kube-root-ca.crt\"" Jul 31 15:11:52.400387 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.400249 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"default-dockercfg-lrnqb\"" Jul 31 15:11:52.400387 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.400249 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"rate-test-model-9966b2-kserve-self-signed-certs\"" Jul 31 15:11:52.400387 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.400308 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"openshift-service-ca.crt\"" Jul 31 15:11:52.406241 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.406220 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv"] Jul 31 15:11:52.478711 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.478678 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:11:52.504914 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.504884 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-model-cache\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.505048 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.504951 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7vb\" (UniqueName: \"kubernetes.io/projected/7017f92d-1500-4f96-9dec-616fd52fbc87-kube-api-access-fq7vb\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.505048 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.504991 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7017f92d-1500-4f96-9dec-616fd52fbc87-tls-certs\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.505123 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.505074 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-tmp-dir\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.505123 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.505112 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-dshm\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.505196 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.505149 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-home\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606316 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606275 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-tmp-dir\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606503 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606325 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-dshm\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606503 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606373 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-home\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606503 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606396 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-model-cache\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606503 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606423 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7vb\" (UniqueName: \"kubernetes.io/projected/7017f92d-1500-4f96-9dec-616fd52fbc87-kube-api-access-fq7vb\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606503 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606467 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7017f92d-1500-4f96-9dec-616fd52fbc87-tls-certs\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606782 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606728 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-tmp-dir\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606782 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606758 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-model-cache\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.606896 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.606811 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-home\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.608805 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.608774 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-dshm\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.608945 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.608923 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7017f92d-1500-4f96-9dec-616fd52fbc87-tls-certs\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.614225 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.614202 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7vb\" (UniqueName: \"kubernetes.io/projected/7017f92d-1500-4f96-9dec-616fd52fbc87-kube-api-access-fq7vb\") pod \"rate-test-model-9966b2-kserve-54b6c4f44c-s74vv\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.706708 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.706643 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:11:52.841163 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.841106 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv"] Jul 31 15:11:52.843639 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:11:52.843612 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7017f92d_1500_4f96_9dec_616fd52fbc87.slice/crio-dd9461965383a2a574f3821532bab6cacd6614346a9b03b05d4372f62652068a WatchSource:0}: Error finding container dd9461965383a2a574f3821532bab6cacd6614346a9b03b05d4372f62652068a: Status 404 returned error can't find the container with id dd9461965383a2a574f3821532bab6cacd6614346a9b03b05d4372f62652068a Jul 31 15:11:52.928959 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.928928 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" event={"ID":"7017f92d-1500-4f96-9dec-616fd52fbc87","Type":"ContainerStarted","Data":"02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321"} Jul 31 15:11:52.929115 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.928970 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" event={"ID":"7017f92d-1500-4f96-9dec-616fd52fbc87","Type":"ContainerStarted","Data":"dd9461965383a2a574f3821532bab6cacd6614346a9b03b05d4372f62652068a"} Jul 31 15:11:52.946843 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:11:52.946796 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" podStartSLOduration=0.946781856 podStartE2EDuration="946.781856ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:52.945037957 +0000 UTC m=+3578.453417024" watchObservedRunningTime="2026-07-31 15:11:52.946781856 +0000 UTC m=+3578.455160921" Jul 31 15:12:02.706783 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:12:02.706733 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:12:02.706783 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:12:02.706792 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:12:02.719963 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:12:02.719932 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:12:03.000705 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:12:03.000621 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:12:22.816704 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:12:22.816671 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:36.233306 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:36.233263 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv"] Jul 31 15:13:36.246126 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:36.246099 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:37.213576 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:37.213534 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:51.252880 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.252839 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" podUID="7017f92d-1500-4f96-9dec-616fd52fbc87" containerName="main" containerID="cri-o://02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321" gracePeriod=45 Jul 31 15:13:51.507954 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.507868 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:13:51.569060 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.569025 2558 generic.go:356] "Generic (PLEG): container finished" podID="7017f92d-1500-4f96-9dec-616fd52fbc87" containerID="02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321" exitCode=0 Jul 31 15:13:51.569225 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.569067 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" Jul 31 15:13:51.569225 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.569105 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" event={"ID":"7017f92d-1500-4f96-9dec-616fd52fbc87","Type":"ContainerDied","Data":"02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321"} Jul 31 15:13:51.569225 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.569151 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv" event={"ID":"7017f92d-1500-4f96-9dec-616fd52fbc87","Type":"ContainerDied","Data":"dd9461965383a2a574f3821532bab6cacd6614346a9b03b05d4372f62652068a"} Jul 31 15:13:51.569225 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.569169 2558 scope.go:117] "RemoveContainer" containerID="02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321" Jul 31 15:13:51.578037 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.578019 2558 scope.go:117] "RemoveContainer" containerID="02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321" Jul 31 15:13:51.578291 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:13:51.578262 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321\": container with ID starting with 02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321 not found: ID does not exist" containerID="02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321" Jul 31 15:13:51.578368 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.578294 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321"} err="failed to get container status \"02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321\": rpc error: code = NotFound desc = could not find container \"02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321\": container with ID starting with 02d75062f5f2bcedc9f0b2ab3afdf57d6671bc2b9658d8c2e65786a7276ff321 not found: ID does not exist" Jul 31 15:13:51.629525 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629499 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq7vb\" (UniqueName: \"kubernetes.io/projected/7017f92d-1500-4f96-9dec-616fd52fbc87-kube-api-access-fq7vb\") pod \"7017f92d-1500-4f96-9dec-616fd52fbc87\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " Jul 31 15:13:51.629664 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629540 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-tmp-dir\") pod \"7017f92d-1500-4f96-9dec-616fd52fbc87\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " Jul 31 15:13:51.629664 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629581 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-home\") pod \"7017f92d-1500-4f96-9dec-616fd52fbc87\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " Jul 31 15:13:51.629664 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629598 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-model-cache\") pod \"7017f92d-1500-4f96-9dec-616fd52fbc87\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " Jul 31 15:13:51.629664 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629613 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7017f92d-1500-4f96-9dec-616fd52fbc87-tls-certs\") pod \"7017f92d-1500-4f96-9dec-616fd52fbc87\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " Jul 31 15:13:51.629664 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629635 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-dshm\") pod \"7017f92d-1500-4f96-9dec-616fd52fbc87\" (UID: \"7017f92d-1500-4f96-9dec-616fd52fbc87\") " Jul 31 15:13:51.629958 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629932 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-home" (OuterVolumeSpecName: "home") pod "7017f92d-1500-4f96-9dec-616fd52fbc87" (UID: "7017f92d-1500-4f96-9dec-616fd52fbc87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:13:51.629958 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629948 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-model-cache" (OuterVolumeSpecName: "model-cache") pod "7017f92d-1500-4f96-9dec-616fd52fbc87" (UID: "7017f92d-1500-4f96-9dec-616fd52fbc87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:13:51.630065 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.629998 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7017f92d-1500-4f96-9dec-616fd52fbc87" (UID: "7017f92d-1500-4f96-9dec-616fd52fbc87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:13:51.631822 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.631796 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-dshm" (OuterVolumeSpecName: "dshm") pod "7017f92d-1500-4f96-9dec-616fd52fbc87" (UID: "7017f92d-1500-4f96-9dec-616fd52fbc87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 31 15:13:51.631971 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.631862 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7017f92d-1500-4f96-9dec-616fd52fbc87-kube-api-access-fq7vb" (OuterVolumeSpecName: "kube-api-access-fq7vb") pod "7017f92d-1500-4f96-9dec-616fd52fbc87" (UID: "7017f92d-1500-4f96-9dec-616fd52fbc87"). InnerVolumeSpecName "kube-api-access-fq7vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:13:51.631971 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.631925 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7017f92d-1500-4f96-9dec-616fd52fbc87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7017f92d-1500-4f96-9dec-616fd52fbc87" (UID: "7017f92d-1500-4f96-9dec-616fd52fbc87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:13:51.731232 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.731196 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-dshm\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:13:51.731232 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.731225 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq7vb\" (UniqueName: \"kubernetes.io/projected/7017f92d-1500-4f96-9dec-616fd52fbc87-kube-api-access-fq7vb\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:13:51.731422 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.731255 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-tmp-dir\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:13:51.731422 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.731265 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-home\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:13:51.731422 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.731273 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7017f92d-1500-4f96-9dec-616fd52fbc87-model-cache\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:13:51.731422 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.731280 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7017f92d-1500-4f96-9dec-616fd52fbc87-tls-certs\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:13:51.893204 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.893177 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv"] Jul 31 15:13:51.897026 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:51.896987 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/rate-test-model-9966b2-kserve-54b6c4f44c-s74vv"] Jul 31 15:13:52.983837 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:52.983798 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7017f92d-1500-4f96-9dec-616fd52fbc87" path="/var/lib/kubelet/pods/7017f92d-1500-4f96-9dec-616fd52fbc87/volumes" Jul 31 15:13:57.340101 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:57.340065 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:57.674482 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:57.674438 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:57.781398 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:57.781365 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:58.243559 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:58.243526 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:13:58.343577 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:13:58.343541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:00.751150 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:00.751112 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:00.856995 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:00.856960 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:01.812995 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.812956 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72"] Jul 31 15:14:01.813383 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.813348 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7017f92d-1500-4f96-9dec-616fd52fbc87" containerName="main" Jul 31 15:14:01.813383 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.813359 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7017f92d-1500-4f96-9dec-616fd52fbc87" containerName="main" Jul 31 15:14:01.813455 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.813442 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7017f92d-1500-4f96-9dec-616fd52fbc87" containerName="main" Jul 31 15:14:01.816560 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.816538 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:01.819184 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.819163 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-58af04-serving-cert\"" Jul 31 15:14:01.828652 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.828626 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72"] Jul 31 15:14:01.881041 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.881004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:01.929886 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.929854 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223f34d5-dedb-4f5e-b474-80aa3d66bece-maas-api-tls\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:01.930074 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.929958 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzh2g\" (UniqueName: \"kubernetes.io/projected/223f34d5-dedb-4f5e-b474-80aa3d66bece-kube-api-access-bzh2g\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:01.930074 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.929986 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223f34d5-dedb-4f5e-b474-80aa3d66bece-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:01.982437 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:01.982405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:02.030678 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.030640 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223f34d5-dedb-4f5e-b474-80aa3d66bece-maas-api-tls\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:02.030864 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.030731 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzh2g\" (UniqueName: \"kubernetes.io/projected/223f34d5-dedb-4f5e-b474-80aa3d66bece-kube-api-access-bzh2g\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:02.030864 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.030763 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223f34d5-dedb-4f5e-b474-80aa3d66bece-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:02.031477 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.031457 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223f34d5-dedb-4f5e-b474-80aa3d66bece-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:02.033250 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.033221 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223f34d5-dedb-4f5e-b474-80aa3d66bece-maas-api-tls\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:02.039823 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.039803 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzh2g\" (UniqueName: \"kubernetes.io/projected/223f34d5-dedb-4f5e-b474-80aa3d66bece-kube-api-access-bzh2g\") pod \"maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:02.128411 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.128387 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:02.269657 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.269624 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72"] Jul 31 15:14:02.271048 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:14:02.271016 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223f34d5_dedb_4f5e_b474_80aa3d66bece.slice/crio-90bd3625e6cbe8c2d21c0642d2da4d3a175ebbb2696b69486f13da3701ea93a3 WatchSource:0}: Error finding container 90bd3625e6cbe8c2d21c0642d2da4d3a175ebbb2696b69486f13da3701ea93a3: Status 404 returned error can't find the container with id 90bd3625e6cbe8c2d21c0642d2da4d3a175ebbb2696b69486f13da3701ea93a3 Jul 31 15:14:02.272855 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.272838 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 31 15:14:02.350872 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.350837 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:02.445631 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.445544 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:02.476086 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.476051 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227"] Jul 31 15:14:02.481056 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.481033 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:02.484375 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.484349 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-58af04-dockercfg-2h76c\"" Jul 31 15:14:02.484504 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.484377 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-58af04\"" Jul 31 15:14:02.491288 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.491212 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227"] Jul 31 15:14:02.535390 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.535356 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4220bbb-a9a2-4775-a23e-566a22ad15f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227\" (UID: \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:02.535553 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.535467 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vkj\" (UniqueName: \"kubernetes.io/projected/a4220bbb-a9a2-4775-a23e-566a22ad15f7-kube-api-access-t4vkj\") pod \"payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227\" (UID: \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:02.630077 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.630039 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" event={"ID":"223f34d5-dedb-4f5e-b474-80aa3d66bece","Type":"ContainerStarted","Data":"90bd3625e6cbe8c2d21c0642d2da4d3a175ebbb2696b69486f13da3701ea93a3"} Jul 31 15:14:02.636458 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.636427 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vkj\" (UniqueName: \"kubernetes.io/projected/a4220bbb-a9a2-4775-a23e-566a22ad15f7-kube-api-access-t4vkj\") pod \"payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227\" (UID: \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:02.636588 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.636532 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4220bbb-a9a2-4775-a23e-566a22ad15f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227\" (UID: \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:02.637253 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.637222 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4220bbb-a9a2-4775-a23e-566a22ad15f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227\" (UID: \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:02.646997 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.646971 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vkj\" (UniqueName: \"kubernetes.io/projected/a4220bbb-a9a2-4775-a23e-566a22ad15f7-kube-api-access-t4vkj\") pod \"payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227\" (UID: \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:02.795950 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.795858 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:02.942974 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:02.942944 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227"] Jul 31 15:14:02.943675 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:14:02.943653 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4220bbb_a9a2_4775_a23e_566a22ad15f7.slice/crio-5eb1d93ee9ebc1a689277aa3b529a00ee33c0f2941e043a1d67b68ec58920d39 WatchSource:0}: Error finding container 5eb1d93ee9ebc1a689277aa3b529a00ee33c0f2941e043a1d67b68ec58920d39: Status 404 returned error can't find the container with id 5eb1d93ee9ebc1a689277aa3b529a00ee33c0f2941e043a1d67b68ec58920d39 Jul 31 15:14:03.643991 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:03.643953 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" event={"ID":"a4220bbb-a9a2-4775-a23e-566a22ad15f7","Type":"ContainerStarted","Data":"31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b"} Jul 31 15:14:03.644193 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:03.644043 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" event={"ID":"a4220bbb-a9a2-4775-a23e-566a22ad15f7","Type":"ContainerStarted","Data":"5eb1d93ee9ebc1a689277aa3b529a00ee33c0f2941e043a1d67b68ec58920d39"} Jul 31 15:14:03.644193 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:03.644074 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:03.645516 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:03.645489 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" event={"ID":"223f34d5-dedb-4f5e-b474-80aa3d66bece","Type":"ContainerStarted","Data":"7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c"} Jul 31 15:14:03.645626 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:03.645551 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:03.662677 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:03.662634 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" podStartSLOduration=1.6626187940000001 podStartE2EDuration="1.662618794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:03.660832727 +0000 UTC m=+3709.169211794" watchObservedRunningTime="2026-07-31 15:14:03.662618794 +0000 UTC m=+3709.170997860" Jul 31 15:14:03.680196 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:03.680153 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" podStartSLOduration=2.131862955 podStartE2EDuration="2.680140159s" totalImagesPullingTime="548.277204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:03.677800769 +0000 UTC m=+3709.186179836" watchObservedRunningTime="2026-07-31 15:14:03.680140159 +0000 UTC m=+3709.188519224" Jul 31 15:14:03.980016 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:03.979919 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:09.656524 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:09.656495 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:14:14.653871 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:14.653838 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:14:25.769491 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:25.769461 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:26.459052 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:26.459018 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:26.778416 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:26.778326 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:26.971256 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:26.971222 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:29.691756 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:29.691716 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:29.792226 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:29.792192 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:30.915186 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:30.915152 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks"] Jul 31 15:14:30.919324 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:30.919294 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:30.923154 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:30.923128 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1c9d21-serving-cert\"" Jul 31 15:14:30.932610 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:30.932577 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks"] Jul 31 15:14:30.952743 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:30.952710 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:31.103323 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.103281 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8705d-c3e6-426a-9b87-7feb5c22871a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.103513 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.103339 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7cc\" (UniqueName: \"kubernetes.io/projected/97e8705d-c3e6-426a-9b87-7feb5c22871a-kube-api-access-fr7cc\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.103513 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.103435 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.204610 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.204506 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8705d-c3e6-426a-9b87-7feb5c22871a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.204610 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.204566 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7cc\" (UniqueName: \"kubernetes.io/projected/97e8705d-c3e6-426a-9b87-7feb5c22871a-kube-api-access-fr7cc\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.204610 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.204606 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.204851 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:14:31.204738 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-serving-cert: secret "maas-api-e2e-ipp-b-1c9d21-serving-cert" not found Jul 31 15:14:31.204851 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:14:31.204808 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls podName:97e8705d-c3e6-426a-9b87-7feb5c22871a nodeName:}" failed. No retries permitted until 2026-07-31 15:14:31.704788439 +0000 UTC m=+3737.213167483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls") pod "maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" (UID: "97e8705d-c3e6-426a-9b87-7feb5c22871a") : secret "maas-api-e2e-ipp-b-1c9d21-serving-cert" not found Jul 31 15:14:31.205213 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.205193 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8705d-c3e6-426a-9b87-7feb5c22871a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.222383 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.222346 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7cc\" (UniqueName: \"kubernetes.io/projected/97e8705d-c3e6-426a-9b87-7feb5c22871a-kube-api-access-fr7cc\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.498295 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.498205 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:31.603530 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.603494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:31.679031 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.678996 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:14:31.710136 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.710097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.712750 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.712723 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.834816 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.834728 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:31.975812 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:31.975598 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks"] Jul 31 15:14:31.978997 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:14:31.978968 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e8705d_c3e6_426a_9b87_7feb5c22871a.slice/crio-780280c91df73eadcb530938ad1a8202995567e397a5ebc5ee85e2152ef33551 WatchSource:0}: Error finding container 780280c91df73eadcb530938ad1a8202995567e397a5ebc5ee85e2152ef33551: Status 404 returned error can't find the container with id 780280c91df73eadcb530938ad1a8202995567e397a5ebc5ee85e2152ef33551 Jul 31 15:14:32.801919 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:32.801868 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" event={"ID":"97e8705d-c3e6-426a-9b87-7feb5c22871a","Type":"ContainerStarted","Data":"600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f"} Jul 31 15:14:32.802090 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:32.801963 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:32.802090 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:32.801981 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" event={"ID":"97e8705d-c3e6-426a-9b87-7feb5c22871a","Type":"ContainerStarted","Data":"780280c91df73eadcb530938ad1a8202995567e397a5ebc5ee85e2152ef33551"} Jul 31 15:14:32.820024 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:32.819949 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" podStartSLOduration=2.166311606 podStartE2EDuration="2.819933637s" totalImagesPullingTime="653.622031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:32.81839611 +0000 UTC m=+3738.326775176" watchObservedRunningTime="2026-07-31 15:14:32.819933637 +0000 UTC m=+3738.328312699" Jul 31 15:14:38.813475 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:38.813437 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:14:54.709418 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:14:54.709377 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:01.702423 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:01.702390 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mgmqg_ddb0610d-5aa3-463d-a582-a3ba5a58c207/payload-processing/0.log" Jul 31 15:15:02.996701 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:02.996664 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:41.498265 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:41.498161 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227_a4220bbb-a9a2-4775-a23e-566a22ad15f7/payload-processing/0.log" Jul 31 15:15:41.626458 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:41.626426 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mgmqg_ddb0610d-5aa3-463d-a582-a3ba5a58c207/payload-processing/0.log" Jul 31 15:15:42.011638 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:42.011603 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:42.379402 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:42.379371 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:42.700645 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:42.700546 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:45.251990 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:45.251956 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:45.510529 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:45.510438 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.342744 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.342702 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c"] Jul 31 15:15:46.347406 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.347387 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.350368 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.350348 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-39573e-serving-cert\"" Jul 31 15:15:46.358288 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.358265 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c"] Jul 31 15:15:46.377156 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.377118 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.511090 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.511052 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b919c204-9e58-46d3-863b-5f18f3024670-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.511292 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.511155 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsw4\" (UniqueName: \"kubernetes.io/projected/b919c204-9e58-46d3-863b-5f18f3024670-kube-api-access-gnsw4\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.511292 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.511250 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919c204-9e58-46d3-863b-5f18f3024670-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.533815 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.533780 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.612321 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.612222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919c204-9e58-46d3-863b-5f18f3024670-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.612321 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.612274 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b919c204-9e58-46d3-863b-5f18f3024670-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.612321 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.612324 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsw4\" (UniqueName: \"kubernetes.io/projected/b919c204-9e58-46d3-863b-5f18f3024670-kube-api-access-gnsw4\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.612957 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.612930 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919c204-9e58-46d3-863b-5f18f3024670-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.614936 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.614892 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b919c204-9e58-46d3-863b-5f18f3024670-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.620761 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.620741 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsw4\" (UniqueName: \"kubernetes.io/projected/b919c204-9e58-46d3-863b-5f18f3024670-kube-api-access-gnsw4\") pod \"maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.661773 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.661743 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:46.808127 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.808096 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c"] Jul 31 15:15:46.810226 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:15:46.809676 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb919c204_9e58_46d3_863b_5f18f3024670.slice/crio-de51509785c79176178ded110d9c4069e337d9c7a94be1102e4a4a6903d5c447 WatchSource:0}: Error finding container de51509785c79176178ded110d9c4069e337d9c7a94be1102e4a4a6903d5c447: Status 404 returned error can't find the container with id de51509785c79176178ded110d9c4069e337d9c7a94be1102e4a4a6903d5c447 Jul 31 15:15:46.857709 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.857676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.952899 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.952866 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:15:46.957998 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.957967 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph"] Jul 31 15:15:46.962634 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.962619 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:46.966156 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.966135 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-39573e-dockercfg-khwwj\"" Jul 31 15:15:46.966493 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.966475 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-39573e\"" Jul 31 15:15:46.973461 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:46.973431 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph"] Jul 31 15:15:47.117231 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.117196 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph\" (UID: \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:47.117442 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.117284 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxjc\" (UniqueName: \"kubernetes.io/projected/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-kube-api-access-scxjc\") pod \"payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph\" (UID: \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:47.218393 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.218326 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph\" (UID: \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:47.218393 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.218379 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxjc\" (UniqueName: \"kubernetes.io/projected/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-kube-api-access-scxjc\") pod \"payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph\" (UID: \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:47.219017 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.218992 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph\" (UID: \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:47.226855 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.226825 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" event={"ID":"b919c204-9e58-46d3-863b-5f18f3024670","Type":"ContainerStarted","Data":"cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e"} Jul 31 15:15:47.226993 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.226938 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" event={"ID":"b919c204-9e58-46d3-863b-5f18f3024670","Type":"ContainerStarted","Data":"de51509785c79176178ded110d9c4069e337d9c7a94be1102e4a4a6903d5c447"} Jul 31 15:15:47.227071 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.227061 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:47.228379 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.228360 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxjc\" (UniqueName: \"kubernetes.io/projected/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-kube-api-access-scxjc\") pod \"payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph\" (UID: \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:47.243969 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.243887 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" podStartSLOduration=0.915098481 podStartE2EDuration="1.243848188s" totalImagesPullingTime="328.749707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:47.24376155 +0000 UTC m=+3812.752140616" watchObservedRunningTime="2026-07-31 15:15:47.243848188 +0000 UTC m=+3812.752227253" Jul 31 15:15:47.275118 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.275081 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:47.420897 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:47.420855 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph"] Jul 31 15:15:47.423824 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:15:47.423786 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bda57e0_d2e0_4b1f_ba07_586b2ea80dee.slice/crio-5b8e14d7f62b3b50cb678f448346c5ce8fc43fb44f66850058d768dd34552ba0 WatchSource:0}: Error finding container 5b8e14d7f62b3b50cb678f448346c5ce8fc43fb44f66850058d768dd34552ba0: Status 404 returned error can't find the container with id 5b8e14d7f62b3b50cb678f448346c5ce8fc43fb44f66850058d768dd34552ba0 Jul 31 15:15:48.234078 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:48.234031 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" event={"ID":"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee","Type":"ContainerStarted","Data":"8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00"} Jul 31 15:15:48.234294 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:48.234101 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" event={"ID":"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee","Type":"ContainerStarted","Data":"5b8e14d7f62b3b50cb678f448346c5ce8fc43fb44f66850058d768dd34552ba0"} Jul 31 15:15:48.234294 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:48.234215 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:15:48.250354 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:48.250307 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" podStartSLOduration=2.250291038 podStartE2EDuration="2.250291038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:48.249600787 +0000 UTC m=+3813.757979856" watchObservedRunningTime="2026-07-31 15:15:48.250291038 +0000 UTC m=+3813.758670136" Jul 31 15:15:53.240557 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:53.240525 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:15:59.246620 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:15:59.246584 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:16:10.282842 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:10.282804 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:16.786313 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:16.786275 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c"] Jul 31 15:16:16.787298 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:16.786503 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" podUID="b919c204-9e58-46d3-863b-5f18f3024670" containerName="maas-api" containerID="cri-o://cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e" gracePeriod=30 Jul 31 15:16:16.807221 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:16.807192 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:16.853587 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:16.850652 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph"] Jul 31 15:16:16.853587 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:16.851063 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" podUID="9bda57e0-d2e0-4b1f-ba07-586b2ea80dee" containerName="payload-pre-processing" containerID="cri-o://8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00" gracePeriod=30 Jul 31 15:16:16.881849 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:16.881789 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:16.969485 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:16.969409 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:17.028310 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.028269 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:17.047244 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.047181 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:16:17.110138 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.110114 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:16:17.125388 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.125357 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnsw4\" (UniqueName: \"kubernetes.io/projected/b919c204-9e58-46d3-863b-5f18f3024670-kube-api-access-gnsw4\") pod \"b919c204-9e58-46d3-863b-5f18f3024670\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " Jul 31 15:16:17.125533 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.125422 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919c204-9e58-46d3-863b-5f18f3024670-openshift-service-ca\") pod \"b919c204-9e58-46d3-863b-5f18f3024670\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " Jul 31 15:16:17.125533 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.125498 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b919c204-9e58-46d3-863b-5f18f3024670-maas-api-tls\") pod \"b919c204-9e58-46d3-863b-5f18f3024670\" (UID: \"b919c204-9e58-46d3-863b-5f18f3024670\") " Jul 31 15:16:17.126119 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.126066 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b919c204-9e58-46d3-863b-5f18f3024670-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b919c204-9e58-46d3-863b-5f18f3024670" (UID: "b919c204-9e58-46d3-863b-5f18f3024670"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:16:17.128007 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.127971 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b919c204-9e58-46d3-863b-5f18f3024670-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b919c204-9e58-46d3-863b-5f18f3024670" (UID: "b919c204-9e58-46d3-863b-5f18f3024670"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:16:17.128222 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.128196 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b919c204-9e58-46d3-863b-5f18f3024670-kube-api-access-gnsw4" (OuterVolumeSpecName: "kube-api-access-gnsw4") pod "b919c204-9e58-46d3-863b-5f18f3024670" (UID: "b919c204-9e58-46d3-863b-5f18f3024670"). InnerVolumeSpecName "kube-api-access-gnsw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:16:17.226656 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.226620 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-plugins-config-volume\") pod \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\" (UID: \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\") " Jul 31 15:16:17.226656 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.226665 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scxjc\" (UniqueName: \"kubernetes.io/projected/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-kube-api-access-scxjc\") pod \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\" (UID: \"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee\") " Jul 31 15:16:17.226954 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.226939 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnsw4\" (UniqueName: \"kubernetes.io/projected/b919c204-9e58-46d3-863b-5f18f3024670-kube-api-access-gnsw4\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:17.227042 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.226958 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919c204-9e58-46d3-863b-5f18f3024670-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:17.227042 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.226972 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b919c204-9e58-46d3-863b-5f18f3024670-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:17.227145 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.227111 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bda57e0-d2e0-4b1f-ba07-586b2ea80dee" (UID: "9bda57e0-d2e0-4b1f-ba07-586b2ea80dee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:16:17.228763 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.228743 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-kube-api-access-scxjc" (OuterVolumeSpecName: "kube-api-access-scxjc") pod "9bda57e0-d2e0-4b1f-ba07-586b2ea80dee" (UID: "9bda57e0-d2e0-4b1f-ba07-586b2ea80dee"). InnerVolumeSpecName "kube-api-access-scxjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:16:17.328188 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.328095 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxjc\" (UniqueName: \"kubernetes.io/projected/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-kube-api-access-scxjc\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:17.328188 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.328129 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:17.343438 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.343404 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:17.411512 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.411478 2558 generic.go:356] "Generic (PLEG): container finished" podID="b919c204-9e58-46d3-863b-5f18f3024670" containerID="cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e" exitCode=0 Jul 31 15:16:17.411698 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.411529 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" Jul 31 15:16:17.411698 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.411566 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" event={"ID":"b919c204-9e58-46d3-863b-5f18f3024670","Type":"ContainerDied","Data":"cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e"} Jul 31 15:16:17.411698 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.411618 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c" event={"ID":"b919c204-9e58-46d3-863b-5f18f3024670","Type":"ContainerDied","Data":"de51509785c79176178ded110d9c4069e337d9c7a94be1102e4a4a6903d5c447"} Jul 31 15:16:17.411698 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.411645 2558 scope.go:117] "RemoveContainer" containerID="cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e" Jul 31 15:16:17.413086 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.413059 2558 generic.go:356] "Generic (PLEG): container finished" podID="9bda57e0-d2e0-4b1f-ba07-586b2ea80dee" containerID="8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00" exitCode=0 Jul 31 15:16:17.413258 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.413110 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" Jul 31 15:16:17.413258 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.413134 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" event={"ID":"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee","Type":"ContainerDied","Data":"8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00"} Jul 31 15:16:17.413258 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.413193 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph" event={"ID":"9bda57e0-d2e0-4b1f-ba07-586b2ea80dee","Type":"ContainerDied","Data":"5b8e14d7f62b3b50cb678f448346c5ce8fc43fb44f66850058d768dd34552ba0"} Jul 31 15:16:17.421392 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.421374 2558 scope.go:117] "RemoveContainer" containerID="cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e" Jul 31 15:16:17.421838 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:16:17.421789 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e\": container with ID starting with cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e not found: ID does not exist" containerID="cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e" Jul 31 15:16:17.421963 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.421848 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e"} err="failed to get container status \"cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e\": rpc error: code = NotFound desc = could not find container \"cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e\": container with ID starting with cb1b52383c07eed679b36ef9605e94ea13420af8fa953982300613d0acb3bb6e not found: ID does not exist" Jul 31 15:16:17.421963 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.421867 2558 scope.go:117] "RemoveContainer" containerID="8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00" Jul 31 15:16:17.430666 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.430648 2558 scope.go:117] "RemoveContainer" containerID="8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00" Jul 31 15:16:17.430945 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:16:17.430927 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00\": container with ID starting with 8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00 not found: ID does not exist" containerID="8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00" Jul 31 15:16:17.431005 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.430961 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00"} err="failed to get container status \"8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00\": rpc error: code = NotFound desc = could not find container \"8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00\": container with ID starting with 8cde8327aff73fb8d9228091c21f799f25ea08061192c6c55eeed7936a2aac00 not found: ID does not exist" Jul 31 15:16:17.439169 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.439146 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c"] Jul 31 15:16:17.441702 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.441682 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-39573e-b47b545b6-tmg4c"] Jul 31 15:16:17.451550 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.451522 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph"] Jul 31 15:16:17.454710 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:17.454688 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39573e-556df86864-qnrph"] Jul 31 15:16:18.983351 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:18.983315 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bda57e0-d2e0-4b1f-ba07-586b2ea80dee" path="/var/lib/kubelet/pods/9bda57e0-d2e0-4b1f-ba07-586b2ea80dee/volumes" Jul 31 15:16:18.983763 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:18.983725 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b919c204-9e58-46d3-863b-5f18f3024670" path="/var/lib/kubelet/pods/b919c204-9e58-46d3-863b-5f18f3024670/volumes" Jul 31 15:16:23.217002 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:23.216960 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:23.317821 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:23.317788 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:23.475719 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:23.475628 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:23.724022 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:23.723981 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:42.649283 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.649176 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72"] Jul 31 15:16:42.649770 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.649430 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" podUID="223f34d5-dedb-4f5e-b474-80aa3d66bece" containerName="maas-api" containerID="cri-o://7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c" gracePeriod=30 Jul 31 15:16:42.679441 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.679405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:42.705230 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.705202 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227"] Jul 31 15:16:42.705446 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.705422 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" podUID="a4220bbb-a9a2-4775-a23e-566a22ad15f7" containerName="payload-processing" containerID="cri-o://31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b" gracePeriod=30 Jul 31 15:16:42.779317 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.779272 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:42.879815 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.879782 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:42.986218 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.986193 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:16:42.989624 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:42.989598 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:16:43.068003 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.067971 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223f34d5-dedb-4f5e-b474-80aa3d66bece-openshift-service-ca\") pod \"223f34d5-dedb-4f5e-b474-80aa3d66bece\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " Jul 31 15:16:43.068200 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.068046 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223f34d5-dedb-4f5e-b474-80aa3d66bece-maas-api-tls\") pod \"223f34d5-dedb-4f5e-b474-80aa3d66bece\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " Jul 31 15:16:43.068200 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.068063 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4vkj\" (UniqueName: \"kubernetes.io/projected/a4220bbb-a9a2-4775-a23e-566a22ad15f7-kube-api-access-t4vkj\") pod \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\" (UID: \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\") " Jul 31 15:16:43.068200 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.068089 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzh2g\" (UniqueName: \"kubernetes.io/projected/223f34d5-dedb-4f5e-b474-80aa3d66bece-kube-api-access-bzh2g\") pod \"223f34d5-dedb-4f5e-b474-80aa3d66bece\" (UID: \"223f34d5-dedb-4f5e-b474-80aa3d66bece\") " Jul 31 15:16:43.068200 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.068175 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4220bbb-a9a2-4775-a23e-566a22ad15f7-plugins-config-volume\") pod \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\" (UID: \"a4220bbb-a9a2-4775-a23e-566a22ad15f7\") " Jul 31 15:16:43.068433 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.068377 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223f34d5-dedb-4f5e-b474-80aa3d66bece-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "223f34d5-dedb-4f5e-b474-80aa3d66bece" (UID: "223f34d5-dedb-4f5e-b474-80aa3d66bece"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:16:43.068656 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.068626 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4220bbb-a9a2-4775-a23e-566a22ad15f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4220bbb-a9a2-4775-a23e-566a22ad15f7" (UID: "a4220bbb-a9a2-4775-a23e-566a22ad15f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:16:43.070331 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.070303 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223f34d5-dedb-4f5e-b474-80aa3d66bece-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "223f34d5-dedb-4f5e-b474-80aa3d66bece" (UID: "223f34d5-dedb-4f5e-b474-80aa3d66bece"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:16:43.070429 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.070352 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4220bbb-a9a2-4775-a23e-566a22ad15f7-kube-api-access-t4vkj" (OuterVolumeSpecName: "kube-api-access-t4vkj") pod "a4220bbb-a9a2-4775-a23e-566a22ad15f7" (UID: "a4220bbb-a9a2-4775-a23e-566a22ad15f7"). InnerVolumeSpecName "kube-api-access-t4vkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:16:43.070429 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.070363 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223f34d5-dedb-4f5e-b474-80aa3d66bece-kube-api-access-bzh2g" (OuterVolumeSpecName: "kube-api-access-bzh2g") pod "223f34d5-dedb-4f5e-b474-80aa3d66bece" (UID: "223f34d5-dedb-4f5e-b474-80aa3d66bece"). InnerVolumeSpecName "kube-api-access-bzh2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:16:43.176466 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.176374 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/223f34d5-dedb-4f5e-b474-80aa3d66bece-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:43.176466 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.176404 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4vkj\" (UniqueName: \"kubernetes.io/projected/a4220bbb-a9a2-4775-a23e-566a22ad15f7-kube-api-access-t4vkj\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:43.176466 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.176416 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzh2g\" (UniqueName: \"kubernetes.io/projected/223f34d5-dedb-4f5e-b474-80aa3d66bece-kube-api-access-bzh2g\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:43.176466 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.176425 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4220bbb-a9a2-4775-a23e-566a22ad15f7-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:43.176466 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.176433 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/223f34d5-dedb-4f5e-b474-80aa3d66bece-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:16:43.220435 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.220402 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:16:43.561519 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.561421 2558 generic.go:356] "Generic (PLEG): container finished" podID="a4220bbb-a9a2-4775-a23e-566a22ad15f7" containerID="31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b" exitCode=0 Jul 31 15:16:43.561519 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.561488 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" Jul 31 15:16:43.561761 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.561508 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" event={"ID":"a4220bbb-a9a2-4775-a23e-566a22ad15f7","Type":"ContainerDied","Data":"31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b"} Jul 31 15:16:43.561761 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.561589 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227" event={"ID":"a4220bbb-a9a2-4775-a23e-566a22ad15f7","Type":"ContainerDied","Data":"5eb1d93ee9ebc1a689277aa3b529a00ee33c0f2941e043a1d67b68ec58920d39"} Jul 31 15:16:43.561761 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.561621 2558 scope.go:117] "RemoveContainer" containerID="31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b" Jul 31 15:16:43.563034 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.563011 2558 generic.go:356] "Generic (PLEG): container finished" podID="223f34d5-dedb-4f5e-b474-80aa3d66bece" containerID="7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c" exitCode=0 Jul 31 15:16:43.563147 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.563079 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" Jul 31 15:16:43.563147 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.563080 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" event={"ID":"223f34d5-dedb-4f5e-b474-80aa3d66bece","Type":"ContainerDied","Data":"7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c"} Jul 31 15:16:43.563147 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.563137 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72" event={"ID":"223f34d5-dedb-4f5e-b474-80aa3d66bece","Type":"ContainerDied","Data":"90bd3625e6cbe8c2d21c0642d2da4d3a175ebbb2696b69486f13da3701ea93a3"} Jul 31 15:16:43.571790 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.571766 2558 scope.go:117] "RemoveContainer" containerID="31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b" Jul 31 15:16:43.572159 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:16:43.572138 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b\": container with ID starting with 31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b not found: ID does not exist" containerID="31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b" Jul 31 15:16:43.572262 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.572167 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b"} err="failed to get container status \"31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b\": rpc error: code = NotFound desc = could not find container \"31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b\": container with ID starting with 31c37e51abf897d0087b553bb878911e3206ccfac4bdbee36f89d86611ce134b not found: ID does not exist" Jul 31 15:16:43.572262 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.572187 2558 scope.go:117] "RemoveContainer" containerID="7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c" Jul 31 15:16:43.581120 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.581102 2558 scope.go:117] "RemoveContainer" containerID="7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c" Jul 31 15:16:43.581397 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:16:43.581380 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c\": container with ID starting with 7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c not found: ID does not exist" containerID="7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c" Jul 31 15:16:43.581467 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.581402 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c"} err="failed to get container status \"7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c\": rpc error: code = NotFound desc = could not find container \"7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c\": container with ID starting with 7270a394c9d073d2df209643de0d6a0e62140c441df48f1e8355510a547c9c5c not found: ID does not exist" Jul 31 15:16:43.587239 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.587215 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72"] Jul 31 15:16:43.589561 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.589536 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-58af04-7956646b4f-hkt72"] Jul 31 15:16:43.599051 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.599020 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227"] Jul 31 15:16:43.602842 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:43.602820 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-58af04-54b55ddf54-tk227"] Jul 31 15:16:44.983599 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:44.983566 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223f34d5-dedb-4f5e-b474-80aa3d66bece" path="/var/lib/kubelet/pods/223f34d5-dedb-4f5e-b474-80aa3d66bece/volumes" Jul 31 15:16:44.983999 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:44.983962 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4220bbb-a9a2-4775-a23e-566a22ad15f7" path="/var/lib/kubelet/pods/a4220bbb-a9a2-4775-a23e-566a22ad15f7/volumes" Jul 31 15:16:48.239823 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:16:48.239792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:03.625866 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:03.625831 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:03.915143 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:03.915052 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:04.731621 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:04.731582 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:10.328446 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:10.328407 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:10.412351 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:10.412317 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:10.616604 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:10.616574 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:10.860574 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:10.860538 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:21.369035 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.368514 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks"] Jul 31 15:17:21.369035 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.368834 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" podUID="97e8705d-c3e6-426a-9b87-7feb5c22871a" containerName="maas-api" containerID="cri-o://600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f" gracePeriod=30 Jul 31 15:17:21.376462 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.376428 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:21.451207 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.451172 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:21.524384 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.524337 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:21.622220 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.622157 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:17:21.635046 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.634995 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:21.745918 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.745884 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr7cc\" (UniqueName: \"kubernetes.io/projected/97e8705d-c3e6-426a-9b87-7feb5c22871a-kube-api-access-fr7cc\") pod \"97e8705d-c3e6-426a-9b87-7feb5c22871a\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " Jul 31 15:17:21.746105 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.745953 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8705d-c3e6-426a-9b87-7feb5c22871a-openshift-service-ca\") pod \"97e8705d-c3e6-426a-9b87-7feb5c22871a\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " Jul 31 15:17:21.746105 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.745985 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls\") pod \"97e8705d-c3e6-426a-9b87-7feb5c22871a\" (UID: \"97e8705d-c3e6-426a-9b87-7feb5c22871a\") " Jul 31 15:17:21.746342 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.746318 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e8705d-c3e6-426a-9b87-7feb5c22871a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97e8705d-c3e6-426a-9b87-7feb5c22871a" (UID: "97e8705d-c3e6-426a-9b87-7feb5c22871a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:21.748106 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.748075 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e8705d-c3e6-426a-9b87-7feb5c22871a-kube-api-access-fr7cc" (OuterVolumeSpecName: "kube-api-access-fr7cc") pod "97e8705d-c3e6-426a-9b87-7feb5c22871a" (UID: "97e8705d-c3e6-426a-9b87-7feb5c22871a"). InnerVolumeSpecName "kube-api-access-fr7cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:21.748185 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.748110 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97e8705d-c3e6-426a-9b87-7feb5c22871a" (UID: "97e8705d-c3e6-426a-9b87-7feb5c22871a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:17:21.776314 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.776286 2558 generic.go:356] "Generic (PLEG): container finished" podID="97e8705d-c3e6-426a-9b87-7feb5c22871a" containerID="600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f" exitCode=0 Jul 31 15:17:21.776442 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.776339 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" Jul 31 15:17:21.776442 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.776372 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" event={"ID":"97e8705d-c3e6-426a-9b87-7feb5c22871a","Type":"ContainerDied","Data":"600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f"} Jul 31 15:17:21.776442 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.776433 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks" event={"ID":"97e8705d-c3e6-426a-9b87-7feb5c22871a","Type":"ContainerDied","Data":"780280c91df73eadcb530938ad1a8202995567e397a5ebc5ee85e2152ef33551"} Jul 31 15:17:21.776545 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.776463 2558 scope.go:117] "RemoveContainer" containerID="600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f" Jul 31 15:17:21.786360 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.786339 2558 scope.go:117] "RemoveContainer" containerID="600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f" Jul 31 15:17:21.786624 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:17:21.786597 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f\": container with ID starting with 600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f not found: ID does not exist" containerID="600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f" Jul 31 15:17:21.786675 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.786625 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f"} err="failed to get container status \"600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f\": rpc error: code = NotFound desc = could not find container \"600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f\": container with ID starting with 600999e6609e6b36b7f61f1391810b4b6105cf5a2525f52cba73b374b5b75a6f not found: ID does not exist" Jul 31 15:17:21.799280 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.799247 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks"] Jul 31 15:17:21.802984 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.802964 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c9d21-57599b84f6-5bxks"] Jul 31 15:17:21.847348 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.847322 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr7cc\" (UniqueName: \"kubernetes.io/projected/97e8705d-c3e6-426a-9b87-7feb5c22871a-kube-api-access-fr7cc\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:21.847348 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.847344 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e8705d-c3e6-426a-9b87-7feb5c22871a-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:21.847483 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.847356 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e8705d-c3e6-426a-9b87-7feb5c22871a-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:21.978323 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:21.978238 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:22.983819 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:22.983784 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e8705d-c3e6-426a-9b87-7feb5c22871a" path="/var/lib/kubelet/pods/97e8705d-c3e6-426a-9b87-7feb5c22871a/volumes" Jul 31 15:17:27.909538 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:27.909496 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:28.010106 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:28.010068 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:28.232438 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:28.232318 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:28.473742 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:28.473702 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:39.128154 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.128121 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx"] Jul 31 15:17:39.128605 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.128346 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" podUID="9ae131da-a85a-4d47-8e5f-35506750dcbd" containerName="maas-api" containerID="cri-o://dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c" gracePeriod=30 Jul 31 15:17:39.151132 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.151102 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:39.247684 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.247656 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:39.321645 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.321610 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:39.380570 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.380484 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:39.388594 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.388565 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:17:39.528434 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.528393 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4frc\" (UniqueName: \"kubernetes.io/projected/9ae131da-a85a-4d47-8e5f-35506750dcbd-kube-api-access-s4frc\") pod \"9ae131da-a85a-4d47-8e5f-35506750dcbd\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " Jul 31 15:17:39.528611 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.528491 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae131da-a85a-4d47-8e5f-35506750dcbd-openshift-service-ca\") pod \"9ae131da-a85a-4d47-8e5f-35506750dcbd\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " Jul 31 15:17:39.528611 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.528519 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls\") pod \"9ae131da-a85a-4d47-8e5f-35506750dcbd\" (UID: \"9ae131da-a85a-4d47-8e5f-35506750dcbd\") " Jul 31 15:17:39.528824 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.528799 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae131da-a85a-4d47-8e5f-35506750dcbd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ae131da-a85a-4d47-8e5f-35506750dcbd" (UID: "9ae131da-a85a-4d47-8e5f-35506750dcbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:39.530644 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.530612 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ae131da-a85a-4d47-8e5f-35506750dcbd" (UID: "9ae131da-a85a-4d47-8e5f-35506750dcbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 31 15:17:39.530787 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.530762 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae131da-a85a-4d47-8e5f-35506750dcbd-kube-api-access-s4frc" (OuterVolumeSpecName: "kube-api-access-s4frc") pod "9ae131da-a85a-4d47-8e5f-35506750dcbd" (UID: "9ae131da-a85a-4d47-8e5f-35506750dcbd"). InnerVolumeSpecName "kube-api-access-s4frc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:39.630124 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.630076 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4frc\" (UniqueName: \"kubernetes.io/projected/9ae131da-a85a-4d47-8e5f-35506750dcbd-kube-api-access-s4frc\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:39.630124 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.630116 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ae131da-a85a-4d47-8e5f-35506750dcbd-openshift-service-ca\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:39.630124 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.630129 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ae131da-a85a-4d47-8e5f-35506750dcbd-maas-api-tls\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:39.725009 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.724882 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:39.872599 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.872563 2558 generic.go:356] "Generic (PLEG): container finished" podID="9ae131da-a85a-4d47-8e5f-35506750dcbd" containerID="dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c" exitCode=0 Jul 31 15:17:39.872758 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.872610 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" Jul 31 15:17:39.872758 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.872647 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" event={"ID":"9ae131da-a85a-4d47-8e5f-35506750dcbd","Type":"ContainerDied","Data":"dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c"} Jul 31 15:17:39.872758 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.872700 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx" event={"ID":"9ae131da-a85a-4d47-8e5f-35506750dcbd","Type":"ContainerDied","Data":"4ef37e2dc9dff0f0e356bc0e6af548ef94475e4ff981ec083514ac744cab14ed"} Jul 31 15:17:39.872758 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.872726 2558 scope.go:117] "RemoveContainer" containerID="dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c" Jul 31 15:17:39.888276 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.888255 2558 scope.go:117] "RemoveContainer" containerID="dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c" Jul 31 15:17:39.888558 ip-10-0-130-33 kubenswrapper[2558]: E0731 15:17:39.888534 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c\": container with ID starting with dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c not found: ID does not exist" containerID="dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c" Jul 31 15:17:39.888643 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.888562 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c"} err="failed to get container status \"dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c\": rpc error: code = NotFound desc = could not find container \"dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c\": container with ID starting with dbe82b395c64638dda665854258515d95fd2d70b7ffc3dd10447dce8f2d5443c not found: ID does not exist" Jul 31 15:17:39.900699 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.900668 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx"] Jul 31 15:17:39.905383 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:39.905359 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9966b2-86d97cbddf-fdvbx"] Jul 31 15:17:40.983632 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:40.983598 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae131da-a85a-4d47-8e5f-35506750dcbd" path="/var/lib/kubelet/pods/9ae131da-a85a-4d47-8e5f-35506750dcbd/volumes" Jul 31 15:17:45.654156 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:45.654124 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:45.752840 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:45.752810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:45.920519 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:45.920405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:46.155591 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:46.155558 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:56.683223 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.683186 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj"] Jul 31 15:17:56.683771 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.683434 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" podUID="44ad3dc5-27d6-46b8-8d8e-10152795f1b7" containerName="payload-processing" containerID="cri-o://71158d4ce2a7915b29e539b82a62101884e46337ff41efb8d2a5cc5f79eec8b8" gracePeriod=30 Jul 31 15:17:56.686716 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.686673 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh"] Jul 31 15:17:56.687177 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.687139 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" podUID="e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" containerName="payload-pre-processing" containerID="cri-o://bf897e0a4095f12633ece79675b0b02ed1a30946f05d68c12c7efcb4e6541106" gracePeriod=30 Jul 31 15:17:56.968228 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.968199 2558 generic.go:356] "Generic (PLEG): container finished" podID="44ad3dc5-27d6-46b8-8d8e-10152795f1b7" containerID="71158d4ce2a7915b29e539b82a62101884e46337ff41efb8d2a5cc5f79eec8b8" exitCode=0 Jul 31 15:17:56.968364 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.968272 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" event={"ID":"44ad3dc5-27d6-46b8-8d8e-10152795f1b7","Type":"ContainerDied","Data":"71158d4ce2a7915b29e539b82a62101884e46337ff41efb8d2a5cc5f79eec8b8"} Jul 31 15:17:56.968364 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.968328 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" event={"ID":"44ad3dc5-27d6-46b8-8d8e-10152795f1b7","Type":"ContainerDied","Data":"951f93c8a035a1d0cb1c4216a98f38792e6796c6337ceca9d3291d451130119b"} Jul 31 15:17:56.968364 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.968359 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="951f93c8a035a1d0cb1c4216a98f38792e6796c6337ceca9d3291d451130119b" Jul 31 15:17:56.969779 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.969758 2558 generic.go:356] "Generic (PLEG): container finished" podID="e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" containerID="bf897e0a4095f12633ece79675b0b02ed1a30946f05d68c12c7efcb4e6541106" exitCode=0 Jul 31 15:17:56.969875 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.969812 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" event={"ID":"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee","Type":"ContainerDied","Data":"bf897e0a4095f12633ece79675b0b02ed1a30946f05d68c12c7efcb4e6541106"} Jul 31 15:17:56.969875 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.969846 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" event={"ID":"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee","Type":"ContainerDied","Data":"0d83cb1dd2073154b9bf017fb2a41f14197ae5b016271a287572bdfb83ba45ca"} Jul 31 15:17:56.969875 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.969864 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d83cb1dd2073154b9bf017fb2a41f14197ae5b016271a287572bdfb83ba45ca" Jul 31 15:17:56.977449 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.977424 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:17:56.980994 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:56.980969 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:17:57.078866 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.078822 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p52pv\" (UniqueName: \"kubernetes.io/projected/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-kube-api-access-p52pv\") pod \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\" (UID: \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\") " Jul 31 15:17:57.078866 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.078879 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-plugins-config-volume\") pod \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\" (UID: \"44ad3dc5-27d6-46b8-8d8e-10152795f1b7\") " Jul 31 15:17:57.079175 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.078973 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-plugins-config-volume\") pod \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\" (UID: \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\") " Jul 31 15:17:57.079175 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.078994 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84rsh\" (UniqueName: \"kubernetes.io/projected/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-kube-api-access-84rsh\") pod \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\" (UID: \"e0fb15d3-82a5-47e1-833d-a8a1e97e45ee\") " Jul 31 15:17:57.079329 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.079299 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44ad3dc5-27d6-46b8-8d8e-10152795f1b7" (UID: "44ad3dc5-27d6-46b8-8d8e-10152795f1b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:57.079377 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.079299 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" (UID: "e0fb15d3-82a5-47e1-833d-a8a1e97e45ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 31 15:17:57.081125 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.081101 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-kube-api-access-84rsh" (OuterVolumeSpecName: "kube-api-access-84rsh") pod "e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" (UID: "e0fb15d3-82a5-47e1-833d-a8a1e97e45ee"). InnerVolumeSpecName "kube-api-access-84rsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:57.081240 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.081182 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-kube-api-access-p52pv" (OuterVolumeSpecName: "kube-api-access-p52pv") pod "44ad3dc5-27d6-46b8-8d8e-10152795f1b7" (UID: "44ad3dc5-27d6-46b8-8d8e-10152795f1b7"). InnerVolumeSpecName "kube-api-access-p52pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 31 15:17:57.179832 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.179790 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:57.179832 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.179824 2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-plugins-config-volume\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:57.179832 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.179837 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84rsh\" (UniqueName: \"kubernetes.io/projected/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee-kube-api-access-84rsh\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:57.180109 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.179851 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p52pv\" (UniqueName: \"kubernetes.io/projected/44ad3dc5-27d6-46b8-8d8e-10152795f1b7-kube-api-access-p52pv\") on node \"ip-10-0-130-33.ec2.internal\" DevicePath \"\"" Jul 31 15:17:57.923572 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.923530 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:57.975681 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.975601 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh" Jul 31 15:17:57.975835 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:57.975604 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj" Jul 31 15:17:58.003172 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:58.003142 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj"] Jul 31 15:17:58.005395 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:58.005369 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:17:58.007656 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:58.007636 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-79a3ee-659d5d4bbb-lhjcj"] Jul 31 15:17:58.019660 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:58.019625 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh"] Jul 31 15:17:58.032814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:58.032785 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-79a3ee-5bb954bc5b-j46kh"] Jul 31 15:17:58.982998 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:58.982955 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ad3dc5-27d6-46b8-8d8e-10152795f1b7" path="/var/lib/kubelet/pods/44ad3dc5-27d6-46b8-8d8e-10152795f1b7/volumes" Jul 31 15:17:58.983531 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:17:58.983508 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" path="/var/lib/kubelet/pods/e0fb15d3-82a5-47e1-833d-a8a1e97e45ee/volumes" Jul 31 15:18:03.173132 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:03.173093 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:03.273632 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:03.273595 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:03.427427 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:03.427341 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:03.797109 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:03.797014 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:04.151855 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:04.151823 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56bdb4749c-qjp6l_6362da31-9896-4393-920c-28f619a89294/authorino/0.log" Jul 31 15:18:09.851461 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:09.851372 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f846468f4-5p9p9_22bf2228-9058-4118-8898-9b0635d27b3f/manager/0.log" Jul 31 15:18:10.204580 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:10.204553 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-5d4h2_44a0597b-822f-4b83-be89-95054f320116/copy-manifests/0.log" Jul 31 15:18:10.218657 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:10.218629 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-5d4h2_44a0597b-822f-4b83-be89-95054f320116/manager/0.log" Jul 31 15:18:10.847406 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:10.847379 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c89d6c77-5stlh_20df45c3-27b5-4510-883c-3e9354e85fbe/maas-api/0.log" Jul 31 15:18:12.122508 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:12.122471 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56bdb4749c-qjp6l_6362da31-9896-4393-920c-28f619a89294/authorino/0.log" Jul 31 15:18:12.471741 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:12.471658 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-wkc27_ea8d159c-6986-4c03-a7cc-568a99ccf0ce/kuadrant-console-plugin/0.log" Jul 31 15:18:13.442618 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:13.442588 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8b44b796-w7hcj_cb68771c-54df-4f39-9822-851485d40255/kube-auth-proxy/0.log" Jul 31 15:18:13.813890 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:13.813810 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-ktn57_66557854-208e-4687-a3dd-33a89c462995/payload-pre-processing/0.log" Jul 31 15:18:13.935725 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:13.935698 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mgmqg_ddb0610d-5aa3-463d-a582-a3ba5a58c207/payload-processing/0.log" Jul 31 15:18:14.053604 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:14.053579 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-776f6c5f98-h4thz_6bc4182d-9581-47d4-bf1a-3e57ee04ffcb/router/0.log" Jul 31 15:18:14.393780 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:14.393746 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wlg29_7d08f180-d899-4391-98ca-b0e60b6fe968/main/0.log" Jul 31 15:18:14.744816 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:14.744738 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpsxvs_c0503050-f883-4e31-bcce-c0f0bf64c480/main/0.log" Jul 31 15:18:14.982197 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:14.982168 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5bljz_7f7690e4-ba60-49c8-82ef-5ff408c248ac/main/0.log" Jul 31 15:18:15.501784 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:15.501745 2558 scope.go:117] "RemoveContainer" containerID="71158d4ce2a7915b29e539b82a62101884e46337ff41efb8d2a5cc5f79eec8b8" Jul 31 15:18:15.511160 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:15.511142 2558 scope.go:117] "RemoveContainer" containerID="bf897e0a4095f12633ece79675b0b02ed1a30946f05d68c12c7efcb4e6541106" Jul 31 15:18:20.644021 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:20.643982 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:20.742539 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:20.742503 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:23.748695 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:23.748659 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vlwj7_ce208340-fc19-4886-b1ce-1e4d7807dbe2/global-pull-secret-syncer/0.log" Jul 31 15:18:23.801350 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:23.801314 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" Jul 31 15:18:23.902181 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:23.902155 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-33.ec2.internal_e8f44f24510d413518febbf8df137749/haproxy/0.log" Jul 31 15:18:28.760802 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:28.760757 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56bdb4749c-qjp6l_6362da31-9896-4393-920c-28f619a89294/authorino/0.log" Jul 31 15:18:28.863063 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:28.863035 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-wkc27_ea8d159c-6986-4c03-a7cc-568a99ccf0ce/kuadrant-console-plugin/0.log" Jul 31 15:18:30.549374 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.549340 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_97b506e6-b571-409f-a6f2-87e791e96f59/alertmanager/0.log" Jul 31 15:18:30.569699 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.569677 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_97b506e6-b571-409f-a6f2-87e791e96f59/config-reloader/0.log" Jul 31 15:18:30.588957 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.588933 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_97b506e6-b571-409f-a6f2-87e791e96f59/kube-rbac-proxy-web/0.log" Jul 31 15:18:30.607745 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.607726 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_97b506e6-b571-409f-a6f2-87e791e96f59/kube-rbac-proxy/0.log" Jul 31 15:18:30.630893 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.630867 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_97b506e6-b571-409f-a6f2-87e791e96f59/kube-rbac-proxy-metric/0.log" Jul 31 15:18:30.664330 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.664299 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_97b506e6-b571-409f-a6f2-87e791e96f59/prom-label-proxy/0.log" Jul 31 15:18:30.684499 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.684475 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_97b506e6-b571-409f-a6f2-87e791e96f59/init-config-reloader/0.log" Jul 31 15:18:30.761994 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.761967 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-nvwtf_6078041b-bb5e-4da7-ba12-f9224586efdf/kube-state-metrics/0.log" Jul 31 15:18:30.779475 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.779450 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-nvwtf_6078041b-bb5e-4da7-ba12-f9224586efdf/kube-rbac-proxy-main/0.log" Jul 31 15:18:30.803955 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.803868 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-nvwtf_6078041b-bb5e-4da7-ba12-f9224586efdf/kube-rbac-proxy-self/0.log" Jul 31 15:18:30.851769 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.851739 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-f8dk9_d4514041-d510-4074-9f05-d8f68f222e95/monitoring-plugin/0.log" Jul 31 15:18:30.879469 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.879444 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2kfn4_eae9b491-dfd9-4415-9fca-a2c5b4e433ff/node-exporter/0.log" Jul 31 15:18:30.903792 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.903769 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2kfn4_eae9b491-dfd9-4415-9fca-a2c5b4e433ff/kube-rbac-proxy/0.log" Jul 31 15:18:30.927351 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:30.927323 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2kfn4_eae9b491-dfd9-4415-9fca-a2c5b4e433ff/init-textfile/0.log" Jul 31 15:18:31.413566 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:31.413535 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-9t8h4_21efdddc-f93a-4975-8a95-dc2ac28811d9/prometheus-operator/0.log" Jul 31 15:18:31.435246 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:31.435220 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-9t8h4_21efdddc-f93a-4975-8a95-dc2ac28811d9/kube-rbac-proxy/0.log" Jul 31 15:18:31.464828 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:31.464801 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-rk7w5_1a56a565-de68-4e6b-ac45-d7421239d643/prometheus-operator-admission-webhook/0.log" Jul 31 15:18:31.500096 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:31.500074 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bd9997cfc-j4jvh_8af786f7-62f4-4021-b74d-3aa1a00268c1/telemeter-client/0.log" Jul 31 15:18:31.524361 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:31.524337 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bd9997cfc-j4jvh_8af786f7-62f4-4021-b74d-3aa1a00268c1/reload/0.log" Jul 31 15:18:31.546044 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:31.546021 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bd9997cfc-j4jvh_8af786f7-62f4-4021-b74d-3aa1a00268c1/kube-rbac-proxy/0.log" Jul 31 15:18:33.004713 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:33.004687 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-xwfbh_b85ca420-fd83-4ed0-90b6-609395bc209c/networking-console-plugin/0.log" Jul 31 15:18:34.211030 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.210999 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-ddb88b868-6c9h7_2b9661f8-768f-44b3-917f-607859268da5/console/0.log" Jul 31 15:18:34.240774 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.240747 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42"] Jul 31 15:18:34.241137 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241124 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e8705d-c3e6-426a-9b87-7feb5c22871a" containerName="maas-api" Jul 31 15:18:34.241183 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241139 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e8705d-c3e6-426a-9b87-7feb5c22871a" containerName="maas-api" Jul 31 15:18:34.241183 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241152 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223f34d5-dedb-4f5e-b474-80aa3d66bece" containerName="maas-api" Jul 31 15:18:34.241183 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241158 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="223f34d5-dedb-4f5e-b474-80aa3d66bece" containerName="maas-api" Jul 31 15:18:34.241183 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241167 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" containerName="payload-pre-processing" Jul 31 15:18:34.241183 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241172 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" containerName="payload-pre-processing" Jul 31 15:18:34.241183 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241178 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bda57e0-d2e0-4b1f-ba07-586b2ea80dee" containerName="payload-pre-processing" Jul 31 15:18:34.241183 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241183 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bda57e0-d2e0-4b1f-ba07-586b2ea80dee" containerName="payload-pre-processing" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241191 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4220bbb-a9a2-4775-a23e-566a22ad15f7" containerName="payload-processing" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241196 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4220bbb-a9a2-4775-a23e-566a22ad15f7" containerName="payload-processing" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241204 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b919c204-9e58-46d3-863b-5f18f3024670" containerName="maas-api" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241216 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b919c204-9e58-46d3-863b-5f18f3024670" containerName="maas-api" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241225 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae131da-a85a-4d47-8e5f-35506750dcbd" containerName="maas-api" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241230 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae131da-a85a-4d47-8e5f-35506750dcbd" containerName="maas-api" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241242 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ad3dc5-27d6-46b8-8d8e-10152795f1b7" containerName="payload-processing" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241249 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ad3dc5-27d6-46b8-8d8e-10152795f1b7" containerName="payload-processing" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241303 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e8705d-c3e6-426a-9b87-7feb5c22871a" containerName="maas-api" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241311 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b919c204-9e58-46d3-863b-5f18f3024670" containerName="maas-api" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241318 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ad3dc5-27d6-46b8-8d8e-10152795f1b7" containerName="payload-processing" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241325 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="223f34d5-dedb-4f5e-b474-80aa3d66bece" containerName="maas-api" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241330 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0fb15d3-82a5-47e1-833d-a8a1e97e45ee" containerName="payload-pre-processing" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241337 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae131da-a85a-4d47-8e5f-35506750dcbd" containerName="maas-api" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241342 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4220bbb-a9a2-4775-a23e-566a22ad15f7" containerName="payload-processing" Jul 31 15:18:34.241412 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.241349 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bda57e0-d2e0-4b1f-ba07-586b2ea80dee" containerName="payload-pre-processing" Jul 31 15:18:34.244457 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.244437 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.247518 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.247496 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7w47\"/\"openshift-service-ca.crt\"" Jul 31 15:18:34.247642 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.247561 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l7w47\"/\"default-dockercfg-8djw8\"" Jul 31 15:18:34.247642 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.247579 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7w47\"/\"kube-root-ca.crt\"" Jul 31 15:18:34.262313 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.259319 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42"] Jul 31 15:18:34.326479 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.326438 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-podres\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.326479 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.326481 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-proc\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.326707 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.326579 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-sys\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.326707 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.326616 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-lib-modules\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.326707 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.326636 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rzh\" (UniqueName: \"kubernetes.io/projected/7909b68c-8014-4e3f-9936-b60347fa3733-kube-api-access-64rzh\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427544 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427496 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-podres\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427544 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427551 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-proc\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427616 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-sys\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427647 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-lib-modules\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427683 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-podres\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427673 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rzh\" (UniqueName: \"kubernetes.io/projected/7909b68c-8014-4e3f-9936-b60347fa3733-kube-api-access-64rzh\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427706 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-proc\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427710 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-sys\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.427814 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.427784 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7909b68c-8014-4e3f-9936-b60347fa3733-lib-modules\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.437897 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.437868 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rzh\" (UniqueName: \"kubernetes.io/projected/7909b68c-8014-4e3f-9936-b60347fa3733-kube-api-access-64rzh\") pod \"perf-node-gather-daemonset-krx42\" (UID: \"7909b68c-8014-4e3f-9936-b60347fa3733\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.556220 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.556115 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:34.690997 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:34.690969 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42"] Jul 31 15:18:34.692545 ip-10-0-130-33 kubenswrapper[2558]: W0731 15:18:34.692522 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7909b68c_8014_4e3f_9936_b60347fa3733.slice/crio-9a77424d25aa993a467705c66b9b15c52bc59c67da8302f5e7483b223d907335 WatchSource:0}: Error finding container 9a77424d25aa993a467705c66b9b15c52bc59c67da8302f5e7483b223d907335: Status 404 returned error can't find the container with id 9a77424d25aa993a467705c66b9b15c52bc59c67da8302f5e7483b223d907335 Jul 31 15:18:35.167053 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:35.167016 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" event={"ID":"7909b68c-8014-4e3f-9936-b60347fa3733","Type":"ContainerStarted","Data":"c8dd1133b57b39af6e498968f2098a644b8240c732ce5e4dc9d88b5dc67e56e7"} Jul 31 15:18:35.167237 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:35.167079 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" event={"ID":"7909b68c-8014-4e3f-9936-b60347fa3733","Type":"ContainerStarted","Data":"9a77424d25aa993a467705c66b9b15c52bc59c67da8302f5e7483b223d907335"} Jul 31 15:18:35.167237 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:35.167108 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:35.183359 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:35.183311 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" podStartSLOduration=1.183298349 podStartE2EDuration="1.183298349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:35.182317494 +0000 UTC m=+3980.690696561" watchObservedRunningTime="2026-07-31 15:18:35.183298349 +0000 UTC m=+3980.691677414" Jul 31 15:18:36.230726 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:36.230686 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nwlqc_697d6e95-2340-45cf-8c34-626bd16dbd73/dns/0.log" Jul 31 15:18:36.249314 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:36.249285 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nwlqc_697d6e95-2340-45cf-8c34-626bd16dbd73/kube-rbac-proxy/0.log" Jul 31 15:18:36.270094 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:36.270069 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8wjhs_809b2748-9aad-4e40-864e-2252739a4f03/dns-node-resolver/0.log" Jul 31 15:18:36.927433 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:36.927401 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kd9wm_4c0cf086-f032-4b04-adcb-23df0705d67c/node-ca/0.log" Jul 31 15:18:38.090072 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:38.090042 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8b44b796-w7hcj_cb68771c-54df-4f39-9822-851485d40255/kube-auth-proxy/0.log" Jul 31 15:18:38.182947 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:38.182888 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-ktn57_66557854-208e-4687-a3dd-33a89c462995/payload-pre-processing/0.log" Jul 31 15:18:38.209653 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:38.209624 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mgmqg_ddb0610d-5aa3-463d-a582-a3ba5a58c207/payload-processing/0.log" Jul 31 15:18:38.238376 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:38.238350 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-776f6c5f98-h4thz_6bc4182d-9581-47d4-bf1a-3e57ee04ffcb/router/0.log" Jul 31 15:18:39.091827 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:39.091794 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bzp46_eabbfc47-226f-4ed0-a0aa-a495528d9f29/serve-healthcheck-canary/0.log" Jul 31 15:18:39.718791 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:39.718757 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l5rch_cc3215d1-396d-4cd5-973c-9332b7493c42/kube-rbac-proxy/0.log" Jul 31 15:18:39.738270 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:39.738246 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l5rch_cc3215d1-396d-4cd5-973c-9332b7493c42/exporter/0.log" Jul 31 15:18:39.758022 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:39.757992 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l5rch_cc3215d1-396d-4cd5-973c-9332b7493c42/extractor/0.log" Jul 31 15:18:41.183302 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:41.183273 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-krx42" Jul 31 15:18:42.518024 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:42.517992 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f846468f4-5p9p9_22bf2228-9058-4118-8898-9b0635d27b3f/manager/0.log" Jul 31 15:18:42.765292 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:42.765260 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-5d4h2_44a0597b-822f-4b83-be89-95054f320116/manager/0.log" Jul 31 15:18:42.785950 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:42.785863 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-5d4h2_44a0597b-822f-4b83-be89-95054f320116/copy-manifests/0.log" Jul 31 15:18:44.378972 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:44.378941 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58f6dff6d5-rdmpb_77394d7d-b35b-4bb9-b391-fa9e60a73588/manager/0.log" Jul 31 15:18:44.967845 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:44.967817 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8b44b796-w7hcj_cb68771c-54df-4f39-9822-851485d40255/kube-auth-proxy/0.log" Jul 31 15:18:45.046626 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:45.046592 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-ktn57_66557854-208e-4687-a3dd-33a89c462995/payload-pre-processing/0.log" Jul 31 15:18:45.074079 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:45.074036 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mgmqg_ddb0610d-5aa3-463d-a582-a3ba5a58c207/payload-processing/0.log" Jul 31 15:18:45.100475 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:45.100443 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-776f6c5f98-h4thz_6bc4182d-9581-47d4-bf1a-3e57ee04ffcb/router/0.log" Jul 31 15:18:46.749441 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:46.749409 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8b44b796-w7hcj_cb68771c-54df-4f39-9822-851485d40255/kube-auth-proxy/0.log" Jul 31 15:18:46.822553 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:46.822523 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-ktn57_66557854-208e-4687-a3dd-33a89c462995/payload-pre-processing/0.log" Jul 31 15:18:46.843704 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:46.843673 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mgmqg_ddb0610d-5aa3-463d-a582-a3ba5a58c207/payload-processing/0.log" Jul 31 15:18:46.857265 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:46.857233 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-776f6c5f98-h4thz_6bc4182d-9581-47d4-bf1a-3e57ee04ffcb/router/0.log" Jul 31 15:18:46.948632 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:46.948603 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:47.005876 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:47.005793 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5t4bh_9b6b34e3-8489-4f7e-aee5-df61d5cf87cc/konnectivity-agent/0.log" limit=1024 Jul 31 15:18:50.819514 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:50.819485 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4gd4d_d57b476a-2d1a-4caf-be2c-ea4677ca3602/kube-multus/0.log" Jul 31 15:18:50.841419 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:50.841395 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8tpw_70381455-7d7d-4682-9ad3-0c19ed448194/kube-multus-additional-cni-plugins/0.log" Jul 31 15:18:50.860488 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:50.860463 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8tpw_70381455-7d7d-4682-9ad3-0c19ed448194/egress-router-binary-copy/0.log" Jul 31 15:18:50.878807 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:50.878788 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8tpw_70381455-7d7d-4682-9ad3-0c19ed448194/cni-plugins/0.log" Jul 31 15:18:50.896595 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:50.896576 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8tpw_70381455-7d7d-4682-9ad3-0c19ed448194/bond-cni-plugin/0.log" Jul 31 15:18:50.913614 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:50.913591 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8tpw_70381455-7d7d-4682-9ad3-0c19ed448194/routeoverride-cni/0.log" Jul 31 15:18:50.931689 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:50.931669 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8tpw_70381455-7d7d-4682-9ad3-0c19ed448194/whereabouts-cni-bincopy/0.log" Jul 31 15:18:50.949269 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:50.949252 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8tpw_70381455-7d7d-4682-9ad3-0c19ed448194/whereabouts-cni/0.log" Jul 31 15:18:51.380252 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:51.380224 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kdtjx_26920aff-943f-4177-8777-d186d5e99fc3/network-metrics-daemon/0.log" Jul 31 15:18:51.395785 ip-10-0-130-33 kubenswrapper[2558]: I0731 15:18:51.395753 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kdtjx_26920aff-943f-4177-8777-d186d5e99fc3/kube-rbac-proxy/0.log"